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A GENERAL INTRODUCTION TO THE SUSTAINING.EFFECTS STUDY 

» , AND 

^ . AN OVERVIEW OF THE BRESfENT REPORT- . 



"GENERAL INTRODUCTION 

In response to questions about education policies, SDC is studying compensatory educa- 
tion (CBf; its nature, quantity, and environment; its sustained effects; and its generality, in a 
large/udy called: The Sustaining Effects Study. This thorough study will.result in a series of 
reports from the following substudies: 

The Longitudinal Study. In the Longitudinal Study, the growt'h of children in .reading, 
t matH, functional literacy, and attitudes toward school were assessed in the fall and spring 

for three consecutive years. The amount and kind of instruction in reading and math 

were also determinedUQr each student, and, in addition, teachers and principals reported 
, on their philosoplfies and practices of .instruction.. Thus, it. was possible not only to 

assess student growth over a three-year period, but to' relate that growth to certain specifics 

of instruction. . >\ 

t 

Th? schools in^the study were drawn from three different groups. The REPRESENTATIVE 
SAMPLE ofscfiools is a- sampfc-carefully. drawn to represent all of the nation's public 
schools that have some of the grades one through six. A second group of schools, rhe 
COMPARISON SAMPLE, is composed of schools that have large proportions of students " 
from paor homes but do not receive special funds to offer CE services. The third group is 
the NOMINATED SAMPLE, composed of schools nominated because their educational 
programs had promise^of being effective for low-achieving students. During the first year.of 
the study, tfata were collected from 328 schools and aboaf 1 18,000 students. 

The Cost/Effeaivene&yStudy. Information was obtained on the resources and services- 
to which each student foas exposed during reading and math instruction. Cost estimates 
were generated on the ba^is of this information. "Because the effectiveness of the instruc- 
tional programs is to be determined as a product of tfife Longitudinal Study, it will be possi- 
% ble to relate the effectiveness of programs to their costs. 

Th$ Participation Srt/dy/The purpose of the Participation Study was to determine the 
relationships among economic status, educational need, and instructional services re- 
ceived The educational achievement of the students and the services they received were 
obtarned in the LQngitudinal Study, and the refined measures of economic status were ob- 
tained in the Participation Study. Visits were made to the homes of over 15,000 randomly 
selected students from the schools in the first-year REPRESENTATIVE SAMPLE. During the 
visits, information was collected on the economic level of the home and on the parents' 
attitude* toward their children's school and learning experience^. Thus, the level of student 
achievement and services^ould be related to the economic level of a student's home. 
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The Summer Study. The Sustaining Effects Study also examined the effectiveness and 
cost-eff?ttiveness of summer-school programs. Information about the summer school 
experiences of stydents was combined with other data. A 'resource-cost' model, 
developed for the regular-year, cost-effectiveness study, was adapted to the needs of the 
summer-sfchool study. ' * 

Successful Practices in High-Poverty Schools. This study is intended to identify and 
describe instructional practices and contexts that appear to be^erfective in raising 5 the 
reading and math achievements of educationally disadvantaged students. In-depth 
observational and interview data were collected from 55 schools that are participating in 
the study. 

THE REPORT SERIES 

Tha major findings of the reports already published are discussed briefly below, along with 
references to the specific reports from the study that have addressed them. 

f ' 

A Description of th& Samples for the Sustaining Effects Study anq( the Nation's Elementary 
Schools. In order to, understand the findings W this study, it is essential to become 
familiar with the characteristics of the samples used and their capabilities of providing 
generalizations to the population of the nation's schools. Technical Report 1 (Hoepfner, 
Zagorski,and Wellisch, 1977) deecribes in detail the samples and how they were formed. It 
also presents the results of a survey of 4,750 public schools with grades in. the 1-6 range by 
projecting the data^to the nation. These projections accurately describe the nation's 
elementary schootsjn termsyof the characteristics of the schools, the kinds of services the 
, sthools provide to students, and the characteristics of the students. The interrelationships 1 
among these characteristics are*also analyzed. 

.Hie different kinds of samples hav^ been explained earlier in this review. Some results 
concerning the characteristics of the nation's public schools are summarized betow:* 

• Enrollment, Urbanism, and Achievement. The total grade 1-6 enrollment in the 
J975-76 school year,wa$~estimated at about 21 million students. There ira moderately 
strong relationship tjetween school enrollment and degree of urbanism, with large 
cities having larger schools than rural areas. Student achievement is related to 
urbanism in a complex way; in general, there are proportionally more large city than 
rural schools that have more than half of their students^ ith achievement levels at least 
one year below grade. 

• 'Compensatory-Education Funds, School Characteristics, and Achievement.* 
About two-thirds of the nation's elementary schools received Title I funds, anct about t 
one-fifth received no compensatory funds from apy sources. There is little relationship 
between the receipt of compensatory funds and the Size 6f a school. However, small- 
city and rural schools tend to receive such funds more frequently than do larg£-city 
schools. As expected, schopls with high concentrations of poor. students tend to 
receive compensatory funds more often than do schools with low concentrations. 
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Similarly, schools with higher percentages of low-achieving students are more likely to 
receive compensatory funds: T # " * 

• Achievement and the Concentration of Poor and Minority Students. There is a 
strong association between percentages of low-achieving students and concentrations- 
of poor and minority students. * 

• School's Grade Sp$n. Generally, the grade span in the school shows weak rela- 
tionships with the size of school, the degree of urbanism, and the concentrations of 
low-achieving, poor, and minority students. 

• Stability of Student Body. Schools tend to have less stability in their student 
bodies as the size of the school increases, and there tends to be less stability in large 
cities. Similarly, stability decreases as concentrations of poor, minority, and low- 
achieving-students increase. 

• Availability of Summer Schools. Fifty-one percent of the nation's schools with 
grades 1-6 have summer-school programs available for tbeir students. Larger schools 
provide summer-school programs more frequently th,an smaller schools. There is prac- 

1 tically no relationship between the availability of summer school and a school's level 
of poverty, minority concentration, or student achievement level. 

A Description of Student Selection for Compensatory Services As It Relate's to Economic 
Status antf Academic Achievement. The Education Amendments of 1974 require .several 
studies to inform Qpngress concerning who dpes and who does not receive Title I services 
and how selection for.siich services is related to theeconomic status of the family -and the 
academic performance of the child .(Section 417 of the General Education Provision Act). 
In addition, th'e federal program administrators want to know the differences between 
the services received by economically and educationally deprived children and those by 
non-deprived children, and the relationship between academic achievement and the 
home environment. 

These questions were addressed in TechnicafcReports.2 (Breglio, Hinckley, and Beal, 1978), 
3 (Hinckley, Beal, and Breglio, 1978),, and 4 (Hinckley, Ed.; Beal, Breglio, Haertel, and 
Wiley, 1979). A brief summary of answers to the questions is provided below: 

• About 29 percent of poor students participate in Title I compared to about 1 1 percent 
k bf the non-poor (Report 2). Looking at CE in ggheral, about 40 percent of the poor 

students arid about 21 percent of the/ion-poor participate. From these findings, we 
can see that proportionally more poor students participate in the services than 
• non-poor. ^ . 

✓ 

• * Using a grade-equivalent metric (one year below expectation for the student's current 

grade) as the definition for* educational disadvantage, about 31 percent of low- 
achieving students participate in Title I, while only 10 percent of regular-achieving 
students do (Report 2). For CE in general, the percentages are 46 for low-achievers and 
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19 for regular-achievers. Among the regular-achievers who participate in CE, many 
score below the national median on achievement tests. 

• Participation rates for Title I and for CE in general, are the highest for students who are 
both economically and educationally disadvantaged (Report 2). Forty-one percent of, 
those students participate in Title I, dnd 54 percent participate in CE in general. 
Participation .rates are uiext highest for students who are educationally but not 
economically needy (26 and 41 percent, respectively), and next' highest for students 

/ economically but not educationally needy (20 and 28 percent, respectively). Only 7 
percent of the students who are neither educationally nor economically needy 
participate in Title I (15 percent forCE in general). These participation rates were inter- 
preted as indicating that the then-current allocation procedures were being complied 
with, and the intentions of the law were being met fairly. : / . 

In comparison to non-poor students, poor students, receive more hours of .instruction 
per year with special teachers, more hours of instruction in medium- and 'small- 
sized groups, fewer hours of independent study, and more non-academic services, 
such as guidance, counseling, health and nutrition (Report 3). The differences are 
even stronger wtlfen poor Title I students are compared to others. Therefore, we can 
conclude that the -distribution of educational service is in line with the intention of 
the laws and regulations. 

• Two aspects of the children's home environments bore significant and consistent 
♦ relations to achievement, the amount of reading done at home and the educational 

attainment of the head of the household. Other variables, sud^s family size, TV- 
watching behavior, and type of living quarters were not consisteiffy related to student ^ 
achievement (Report "4). Although most parents (67 percent) know whether their 
children's schools have special programs for low-achieving students, few (40 percent) 
know of Title I and even fewer know of or participate in local governance of the Title I 
programs. Poor parents, in general, are less involved in their children's educational 
programs, have lower expectations of thejr children's attainments, and give lower 
ratings to the quality of their children's educations, but still perceive^Title I and other 
GE programs as being helpful. % - . 

Description of the Nature of CE Programs, Characteristics of Participating Stude,nts y 
Schools, and Educational Services. The k Participation Study deals almost exclusively with 
what has been called 'selection for CE or Title I seryices,' without examining too closely 
what such programs really, are and how they differ from the programs regularly offered by 
the schools. Before we could draw any conclusions about participatibn in a CE program 
and the educational progress of students, we had to be assured'that there really was a pro- * 
gram that was distinct could be specified in some way, and had a reasonable chance of 
making an impact. As will be seen, not only did we analyze data on the basis 6i program 
participation, but we also considered the actual services received in order to address direct- 
ly the possible differences .between intentions and Actualities. * . 

' Based on the analyses of data obtained from about 81, 50(t students in the Representative 
Sample of .schools,. Technical Report 5 (Wang, Hoepfner, Zagorski, Hemenway, Brown, 
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' Students participating in CE are lower achievers (mean score at the 32nd percentile) 
than non-participants (53rd percentile). Seventy percent of the participants were 
judged by their teachers as needing CE, while only 19 percent of those not par- 
ticipating were so judged. More minority students- participate in CE, proportionately, 
than white students, but participation in C£ has little relationship with" student 
attitudes towards school, early school experiences, summer experiences, or the in- 
volvement of their parents in their educational programs. 

Minority;* poor, and low-achieving students tend to receive more hours of instruction 
in smaller groups and by special teachers, and to receive more non-academic services, 
but their attendance raftes are generally lower, too, so that they do not take maximum 
advantage of the special services provided. 

The useful predictors of whether or not a student is selected to receive CE are his/her 
teacher's judgment of need and participation in CE in the previous year. When these 
variables are considered, achievement scores, non-English language spoken in the 
home/and Economic status contribute little more to the prediction. 

About two-thirds of the students participating in CE in 1975-76 participated in the 
1976-77 school year also. 

CE students in general and Title I students in particular receive more total hours of 
instruction per year than non-CE stud'ents. The CE students also receive more hours of 
instruction from special teachers. Among CE students, 'Title" I students receive the 
greatest number of hours of instruction, more frequently with special teachers, and in 
small instructional groups. There are no significant and consistent differences between 
CE students and;non-CE students with regard to their teachers' instructional subgroup- 
ing practices, the use Qf lesson plans, the extent of individualization of instruction, the 
frequency of feedback, or the assignment of homework. 

Students receive 5 to 9 hours of reading mst ruction pepweek, decreasing steadily with 
higher grades, apd between 5 to 6 hours of math instruction per week, fairly constant 
over all grades? \ 

i 

CE services are delivered during regular instructional hours with different kinds of 
activities for the participants (so that, in effect, they 'miss' some regular instruction 
received by non-participating peers). 

i r 

Title I schools have higher average per-participant CE expenditures in reading and 
math than do schools with other CE 'programs. The average Title I. per-participant 
expenditure is about 35 percent of. the average per-pupil regular (base) expenditure. 

Schools: providing CE generally have higher concentrations of pqor and low-achieving - 
students, and students with less educated parents. These schools have greater 
administrative and instructional control by their districts and have higher staff-to- 
student ratios. I 
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• Schools that select higher percentages of regular-achieving students for CE services 
havfe larger percentages of minority and poor students, probably reflecting their 
tendency for the saturation of CE programs. 

• Most districts use counts of students receiving reduced-price lunches and courts of 
'families receiving AFDC to determine school eligibility for compensafory funds, while 
most schools select students on the basis' of standardized achievement tests, fre- 
quently augmented by teacher judgments. Similar selection criteria are employed by 
non-public schools. 

Cost-Effectiveness Qf Compensatory Education. In its deliberations foVthe reauthoriza- 
tion of Title I, and m annual appropriation hearings, members of Congress also wanted 
information on the effectiveness of the Title I program relative to its cost. While it appears 
eminently sensible to ask the question of cost-effectiveness, it is difficult to provide the 
answers in a manner that will be interpreted correctly. 

In the study of the cost-effectiveness of CE, efforts were made to preclude enigmatic conclu- 
sions and, at the same time, to make cost estimate* on a sounder basis than in the past. In 
Technical Report 6, .Haggart, Klibanoff, Sumner, and Williams (1978) develop and present a 
resource-cost model that translates educational resources for each student into estimates of 
average or standard dollar costs for his/her instructional program. The overall strategy Jor 
estimating costs is tc^*fDv«Je an index that represents the labor-intensity of services without 
being confounded witaNFegiohal price differentials, different accounting methods, etc. 

Using the resource ^sts, CE students in general, and Title I students in particular, were 
found to be offered substantially higher levels of educational resources, and hence more 
costly programs. Participation in CE differentiates the resource costs for services offered 
much more thaa do poverty, achievement level, race, or any other characteristics. . 

In Technical Report 7, Sumner, Klibanoff, and Haggart (1979) related resource costs to 
achievement to arrive at an index of cost-effectiveness. Because of the low-achievement . 
levels of the children participating in CE, and their relatively slow rates of achievement 
growth, the increased costs associated with CE appeared to be misspent (in the same way 
that money for severly ill and terminal patients appeals 'to be not as effectively spent as it is 
for mildly ill patients). 1t is important to point out,, however, that the appearance may not 
tell the trjje story. Because we cannot obtain truly appropriate comparison groups, we do 
not know what would have happened to the achievement growth of the CE students if they 
had not participated. Based on the comparison groups we could form, however, CE 
programs did not appear to have an advantage over regular programs in terms of cost- 
effectiveness. 

The Effectiveness of Summer-School Programs. The study has also examined the result^ 
of attendance at summer school, because members of Congress and program 
administrators wanted to know if such attendance helps prevent the presumed progressive 
academic-deficit of low-achieving students. If attendance at summer school were to have a 
positive academic effect (insofar as that the attendees do not 'fall back' to their achieve- 
ment levels of previous years), then summer programs could be considered as a means of 
sustaining school-year growth. . 

O 
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Technical Report'8 (Klibanoff and Haggart, r980) shows that attendance at summer school 
has little or no effect on academic growth, especially on low-achieving students. Because 
the findings are based on the study of summer schools as they presently exist (and the 
evidence is strong that they do not offer intensive academic experiences), the non-positive 
findings should 6 not be interpreted Js an indictment of summer school, as such, but an 
evaluation of the way they are presently organized and funded. Nevertheless, when 
instructional* services delivered in summer schools were investigated, none seemed par- 
ticularly effectives improving* achievement growth. . / 

In the $ame repfirt, the authors ^Iso addressed the hypothesis of 'summer drop-off/ a 
hypothesis advanced to explain the presumed widening achievement gap between regular 
and CE students. Essentially,.this hyppthesis sfates that CE students lose much more of their 

.previous year's learning during the summer recess than do regular students. Data collected 
in the study fail to support the summer drop-off hypothesis. CE. students do not suffer an 
absolute 'drop-off' (although their achievement growth over the summer is less than that 

; for regular students, as in the school year). In any event, attendance at summer school does 
not have much df, an effect. , j 

Technical Report 9 is a resource book. It identifies all the variables and composites that 
have been delected or devised for use in the Sustaining Effects Study. All measures and 
scales are described and rationalized. In addition, Report 9A serves as a companion 
volume that contains copies of all of the data-collection instruments in the study, except for 
3 few that are constrained by copyright. 

Compensatory 5er\ ices and Educational Dexelopment Jn the School Year. Technical 
Report 10 addresses the effects of compensatory services on student development during 
the school period. It also examines the instructional services and major dimensions of the 
educational process to describe the characteristics of programs that are effective in raising 
^achievement levels. The Analysis is based on the first-year data of the study. Similar 
investigations will con<inue f in subsequent reports. 

Studies Still to be Done. The remaining reports, yet to come from the study, will 
address the general 'effects of educational practices on raising achievement levels, with 
special attention paid to the practices found in CE programs in general and in Title I pro- 
grams ip particular' Impact analyses -will either be based on three-year longitudinal data or 
on in-depth observations and interviews. The extensive achievement data collected from 
'overlapping pbhorjs of students in the three years will be utilized to describe the patterns of" 
education^ grdwtK over the years for various groups of CE and non-CE students. Analyses 
of the thr#>year longitudinal data will allow us to examine the sustained effects of CE and 
help us determine f if fhe presumed phenomenon of gap-widerring between the disadvan- 
taged and non-disadvantaged students indeed exists. 
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This report includes a collection^ substudies that are part of the Sustaining Effects Study 
(SES). The substudies apply selected data to specific policy issues or investigate in greater 
depth certain aspects of our data that were notyully understood. The findings, we believe, 
are of general interest and are of sufficient va I ue\t9*to arrant publication. The substudies are 
organized,. as chapters, into four topics: allocation of Title I funds and services (Part I), 
achievement and compensatory education (Part ll) x measuring student growth (Part III), and 
the effects of attrition. (Part IV). 

ALLOCATION OF TITLE I FUNDS AND SERVICES 

Part l 4 contains fiy e chapters of small studies and analyses especially requested by the 
Department of Education to provide useful informatiorMo the participants in the then 
current Congressional hearings on the reauthorization of THle I in 1978. The first chapter 
addresses poverty concentrations in districts and schools in order to provide estimates of 
the potential incidence^ of 'target areas' and 'concentration grants.* All schools in 'target 
areas' of relatively high pqverty would become automatically eligible to participate in Title" 
I. Districts with very high poverty concentrations are eligible for 'concentration grants.' 
Below, we make projections of tbevnational distribution Of poor districts and schools oyer 
regions and categories of urbantem/The distributions of. schools with poverty concentra- 
tions at or above specified criteria for eligibility are also presented.. 

Percentages of High-Poverty Schools by Region and Urban ism 







Potential Criteria for 




Potential Criteria for 


Region 




Target Areas % , 




Concentration Grants 


Urbanism 


10% + 


20% + 




70% + 




80% + 




Poverty* 


Poverty 4 


Poverty , 


Poverty 




Poverty 


New EngJand 


?.3 


6.7 * 


6.3 


7.4 




5.3 


Metropolitan Northeast 


7.1 


6.7 


6.6 


8.0 




9.1 


Mid-Atlantic 


10.4 


10,6 ' 


10.9 


9.0 




6.4 


Southeast 


"18.4 


22.9 


27.9 


28.3 




28.4 


North Midwest 


18.3 


15.0 


12.6 


10.3 


12.1 


South Central 


13.1 


15.6- 


17.4 


20.5 




21.6 


Central Midwest 


7.3 


5.8 - 


• 4.3 


3.1 




2.9 


North Central 


4.8 


* 3.8 


2.8 


2.2 




2.8 


Pacific Southwest 


9.2 


9.3 


8.7 


'9.2 




8.7 


Pacific Northwest 


4.2 


3.6 ; 


2.4 


2.1 




2.7 


City of over 500,000 


8.3 


10.6 ' 


13.5 


29.2 




35.2 


City of 200,000-500,000 


4.3 


4.9 . 


5.4 


7.4 




• 7.6 


City of 30,000-200,000 ; . 


"10.8 


• 11.1 


12.3 


11.1 




^10.1 


Suburb of city 


9.1 


6.4 * 


4.7 


1,5 




1.2 


City under 50,000 % 


28.2 


27.3 


25.5 


17.8 




14.0 


Rural area near city 


10.5 


8.8 


7.3 


3.6 




2.1 


Rural area not near city 


28.8 


30.8 # 


31.3 • 


29.3 




29.8 



•The'criteria are based on principals' estimates of the percentages of students enrolled in the school who meet 
Title I poverty criteria or are eligible to receive free lunches! * ; "' 
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Because of their poverty, the Southeast ancftouth Central regions, together with large cities 
and remote rural areas, would gain in the number of schools automatically eligible, as the 
required poverty rate increases. On the other hand, if the required poverty rate is' lower, 
the North Midwest and suburbs, small cities, and rural areas near cities would automatical- 
ly gain eligible schools. Higher poverty rates for concentration grants increase'the numbers 
pf schools eligible in the Metropolitan Northeast and the North Midwest regions antf. in 
large cities, but decrease them in the New England and Mid-Atlantic regions and in small 
> cities. (Partially based on these data, the criteria specified in the 1978 reauthorization fcere 
20 percent for target areas and 75 percent for.concentration grants.) * 

tn the second chapter we study the results qf different methods for determining a school's 
poverty level in order to check the validity of data in the SES, and also to pinpoinfany 
weakness in methodology. Five estimates of the concentration of students from 'poor 
families are compared. The estimates, made by different people or using different methods, 
* are highly correlated, but principals were found to make the lowest ones. Teachers' 
estimates and the, application of district allocation formulas were found to be very close to 
the estimates based on the federal allocati6n formula applied to income data from home 
interviews, which we assumed to be the most accurate. No systematic bias in estimates was 
found to be related to such school characteristics as urbanism, enrollment, or minority 
concentration. Principals were found to under estimate school poverty, but they do it so 
consistently that district rankings of schools on that basis can still be expected to conform 
well to Title*! intentions. 

The third chapter is closely related insofar as it provides survey data on the criteria" that 
distncts,use to assess school poverty en route to the determination of Title I eligibility. Over 
78 percent of the' nation's school districts are projected to use counts of free or reduced- 
price lunches, and about 60 percent use AFDC enrollments. Most districts use more than 
one criterion, but we can't tell if they use them jointly in order to converge on 'true' 
estimates of school poverty or to select schools on the basis of,other considerations not 
within the intent of the Title I law. 

The last two chapters provide information on where Tit^TserviGes are being distributed 
We provide tabulations of Title I services by school poverty, average 'school achievement 
minority 'concentration, and enrollment. As would be expected, poorer schools have 
higher rates of Title I participation than less poor schools, but the relationship is not perfect: 
The Title I participation rates' by school poverty level are presented below by region 
and urbanism. 



- Region 

* 




Percentage of Lower 

Poverty Schools 
Participating in Title 1 


Percentage of Higrier 

Poverty Schools 
Participating in Title 1 


New England 
Metropolitan Northeast 
Mid-Atlantic 
Sputheast 
North Midwest 

South Central 
Central Midwest 
North ^Central 
Pacific Southwest 
Pacific Northwest 


* 


52 

i 61 
56 
37 
6t 

39 
64 
42 
14 
50 


79 
84 
.89 

89 

♦ 79 ' 

9 

88 
82 
75 

t 69- 
96 
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Urban ism 


Percentage of Lower 
Poverty Schools^ 
Participating in Title 1 


Percentage of HigheY 
Poverty Schools . 
= j Participating in Titled 


City of over 500,000 
City of 200,00^500,000 " 
Cjty of 50,000-200,009 
Suburb of city 
City under 50,000 
Rural area near city 
• Rural area not near city 


14 
13 
18 
46 

57 
68 
66 


74 
73 

79 ' 

74 

91 

89 

88 


Considering only schools participating in Title 1, the average achievement percentile scores 
for three different kinds of students are presented below. ' 


Participation of Student 


Average 
Reading Percentile 


Average * 
Math Percentile 


' Title 1 students ' 
Other-CE students 
Non«CE students 


26.5 
37.1 

51.8 1 


28.4 
40.1 
' 49.0 



The trend is quite clear that the lowest'achieving students are selected for Title I services. 

Minority children participate in Title I programs in higher proportions than their incidence 
in the population, mostly because they generally attend schools.in high-poverty areas and 
have low achievement levels. Minority participation is highest in the high-poverty schools 
and in the cities. , 

We also briefly examine certain dimensions of the educational services received in the 
name of Title I, and how students are selected to receive them. Title I participants are 
offered the same number of days of instruction per. school year, but have slightly higher 
absence rates. About two-thirds of the CE students continue to receive CE for more than 
one year. While CE students are receiving their CE services] most of their non-CE peers are 
receiving instruction in the same subject, but presumably oqt^ of a compensatory or 
remedial nature. * » 



Finally, we tabulate the incidence of Title I services and services for special students in 
order to disentangle the two largest federal education programs, Title I and P.L, 94-142, in 
terms of their services and effects. The approximate percentages of students with physical, 
psychological, or adjustment problems that are judged to interfere with academic perfor- 
mance by CE status are provided below. 



44 


CE Status of Students 




Percentage of Students 
Judged to Have Problems 




Title I only * 




18' 




Title I and Other CE 




23 


\ 


Other CE only 




20 




NoCE 




9 



9 
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achievement'and compensatory education . * 

Part II contains two chapters that are concerned withrfhow and how well Title J and CE in 
general are distributed at the student level. In the first chapter, we develop several indexes 
of school-level 'targetmg'-the extent to which low achievers are selected for CE participa- 
tion The application of any of the targeting indexes to all schools, without regard to the 
philosophies underlying their CE programs and the wide range of acceptable structuring of 
the programs, results- in softie unfair verdicts. The indexes are compared on the basis of 
their relative merits, but none can be universally recommended. 

The second^ chapter relates line results of scores on achievement tests to teachers' 
judgments of students' needs for CE. We wanted to determine how much agreement there 
was so that, if agreement were%high, the less costly judgments could be encouraged as 
acceptable methods for selecting CE participants. The correlation between scores and 
judgments is about 0.5, overall. If achievement scores are dichotomized near the 35th 
percentile, agreement with teacher judgments is generally maximized, with over 75 per- 
cent of the students categorized consistently as .needing or npt needing CE. 

* 

judgments being what they are, we also studied whctf might influence them so that, even if 
accurate, systematic biases or errors could be pointed out. Although such biases are 
eviderft, they are of very small magnitude. 

MEASURING STUDENT GROWTH 

In/any studytfhat is meant to influence social policy, the measures employed to assess out- 
comes of the programs are critical. In the case of the SES, the outcomes are the growth rates 
of groups of students who do and do not participate in CE programs. Part III describes the 
procedures we employed in the selection and development of the student-level measures 
of academic growth, practical-achievement, and affect. We also investigate issues of testing 
that have been raised by the SES. 

The measures of academic^growth, w^ich are the subject of the first two chapters of Part III, 
are standardized tests and'criteribri-referenced tests of reading and math. Skills in reading 
and math underlie academic achievement in all other areas and ar t e, therefore, the objec- 
tives of most CE programs. After a careful evaluation of all standardized achievement tests, 
the Comprehensive Tests of Basic Skills, Form S, was selected. This test was administered at 
all grades in both the fall and spring. During the first year of the study, each student was N 
tested at the appropriate level and at the next-lower level. The dquble testing allowed for 
the construction of an accurate vertical scale of growth which, in turn, made it possible to 
test at a single functional .level durmg the later years of the study. The test series also under- 
went several analyses in an effort to identify items biased against minority students. Items 
identified as biased were not scored for any of the analyses of the study. 

The measurement needs of the study and the 'theoretical underpinnings and practical 
limitations of criterion-referenced testing were also assessedto-determine the feasibility of 
ejn ploying such an approach in the nationwide study. In general, the practical limitations 
of .criterion-referenced tests and- the level Coi effort needed Jo remedy those limitations 
forced us to conclude that the use of such instruments was not feasible. 
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When the very value of academic skills is called into question, it usually i.s with the suspi- 
cion that such skills do not have much functional value for many individuals. The result of 
this, line of thinking has fcfeen an increased interest in measure^f what has been called 
'functional literacy.' The concept of functional literacy, treated in the third chapter, has 
been expanded in the SES to include functional math skills and thus is called practical 
< achievement. Upon completion of a review of theory and practice in the assessmept of 
functional fiteracy and practical achievement, it was concluded that no existing instrument 
would be appropriate for the students, in the SES. Consequently, an instrument- 
development effort was undertaken that resulted in a reliable and student-acceptable 
instrument that taps practical reading and computing. 

The value of affective growth can be' justified either as a goal in itself or as an instrumental 
goal necessary for academic growth. Either way, measures of affective growth ^ave 
occupied an important position in the batteries of outcome measures developed for the 
study of educational programs. These measures are addressed in the fourth chapter of Part 
III Available measures of aspects of affective growth in children were reviewed with the 
intent of selecting a set of scales as outcome-measures. The Survey of Student Attitudes 
was selected to assess attitudes toward reading and math. Additional items w^re generated 
in a similar format to assess attitudes toward school in general. These three dimen- 
sions^ of attitude appeared in the field test to be relatively independently and reliably 
measured constructs. 

* 

In the fifth chapter we look at out-of-level' testing— a practice whereby very low-achieving 
"'students are tested "with a- level of test designed for students in a lower grade. Average' 
school percentages of scores below the chance level and at the ceiling obtained with at-and 
below-level tests are presented below. 



Average School Percentages of Students Scoring at the Floor and Ceiling 


Test 


At- or Below-Chance Scores 


~ * At- or Near- Perfect Scores 


Level 


Reading Math 


Reading Math 


Grade-Level 


19.2 17.6 


0 8 0.1 


Below- Level 


5.6 ' ' 2.9 

~i — ■ 1 


8.3 % 4.7 ^ 



Jesting vyijh below-level t$sts greatly reduces 'floor' effects, but increases 'ceiling' effects 
♦ somewhat. Out-of-level testing has .much to recommend it on several grounds, but it also 
" 4 presents problems. The pros and cons are discussed in some detail in this chapter. 

In the last chapter we investigate the issue of the 'speededness' of the achievement tests 
.used in the SES. Our special concern is to determine if the speed factor influences scores 
differently for different racial/.ethnic groups. In the event that such ^effects can be 
uncovered, and speed is considered anjrrelevant component of achievement, testing pro- 
cedures in the SES should be modified to eliminate speed so that research findings are not 
irffluenced by it. We found that close to 90 percent of the students attempt the last item of 
regularly timed tests. Test speededness, as a phenomenon, Is found to be most pronounced 
for black students atihe higher grades and less pronounced for brown students. 



THE EFFECTS OF ATTRITION 

■ t 

Part IV is composed of two chapters that address two. distinct issues of attrition from the 
longitudinal study. In the first, we describe the reduction in the SES sample of schools In the 
second year of the study .that resulted from funding cutbacks. Although the retained sample 
was selected with specific study purposes in mind, and could not be assumed to be random 
and representative, tests of the data show its remarkable ability to approximate the" popula- 
tion and the first-year sample. We concluded that the reduced sample will serve the SES 
well, but that the reduction may be expected to have introduced some complications for 
th.e interpretation of some analyses. 

Based on the first full calendar year of the S£S, we document in the second chanjgr the in- 
cidence of attrition of students and compare rates over several characteristicsTsuch as 
achievement level and minority status. The average achievement scores for three groups of 
students are given below. • ' v 









Type of Student 


Average 4 
Reading Percentile 


Average 
Math Percentile 


Students who stay in the study « % 
Students wlttHeft the study 
Students who enter the study late 


49.7 
40.8 
44.7 


50.0 
42.3 

44.5 T 



We can see that the sample of stable students has a higher achievement level" than either 
attrital group. When minority status and poverty are calculated for the three groups, the 
findings are: V 



Type of Student 


Percent That 
Are Minority 


Percent That Receive 
Free or Reduced-Price Meals 


Students who stay in the study 
Students who left the study 
Students-who enter the study late 


' 30.8 
36.9 ' " 
34.0 


1 ~ : - 

39.0 
45.6 * 
50.5 ' 



ln.addition to being lower achievers, students in the attrition groups are more likely to be 
minority students and to receive free* or reduced-price meals. We also study the attrition 
rates by several characteristics' jointly, in order to understand better the major sources. of 
attrition. Finally, we provide some conjectures about the expectable 'influences of the or> 
served attrition on different kinds of analyses and issues in the SES. 
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PART I. ALLOCATION OF TITLE i FUNDS AND SERVICES 

Part I contains five chapters that are collections of analyses that were especially requested 
by the Department of Education to provide information) for the Congressional hearings on 
the reauthorization of Title I in 1978. The first chapter addresses poverty concentrations in 
districts and schools in order to provide estimates 6f-the potential incidence 8 of districts and 
schools with sufficiently high concentrations of poor children so that they can automati- 
cally qualify for special- eligibility, for Title I services' We use a modeling-like process to 
make projections of the national incidence of such districts and schools under different 
poverty-concentration criteria to determine the distribution of such districts and schools 
over regions and ■categories of urbanism. In the second chapter we study the results of 
different methods for defining schooi poverty in the hope not only of checking the validity 
of such. information, but also to insure that weaknesses of method can be pinpointed ancf 
advertised, thus discouraging such methods from common use. The third chapter is closely 
related insofar as it provides survey data on the criteria that districts currently use to assess 
the poverty level of schools. 

• 

The last two chapters providejnformatibn on where Title I services are being provided. We 
provide tabulations o^ Title I services by school poverty, sdtool achievement," minority 
concentration, and enrollment". We also take a brief look at certain dimensions o^those 
services, as well, as how students are selected to receive them. Finally, we tabulate the 
incidence of Title-I services as opposed to services for special (handicapped) students in 
order to disentangle the, services and effects of the two largest federal education programs 
(ESEA Title I and P.L. 94-V42). Thus fe findings of the Sustaining Effects Study will apply more- 
directly td compensatory-education efforts and not to special-education efforts. " 
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CHAPTER 1. SCHOOLS AND DISTRICTS QUALIFYING FOR TITLE I 
UNDER DIFFERENT CRITERIA OF POVERTY CONCENTRATION ' 

Ralph Hoepfner 
Ming-mei Wang 
,v«j Moraye B. Bear 

Henry J*. Zagorski 
* < 
Data from the Sustaining Effects Study (SES) were projected to the nation in 
order to supply the U.S. Department of Education with estimates of the number 
of districts, schools, and students affected by changes in the 1978 amendments to 
• , Title '' es P ecia,, y Ahose changes affecting various eligibility criteria that ' 
' • incorporate She notion of the 'concentration ' of poor children. Alternative defini- 
tions of concentration were<jsed for the projections so that Congress would be 
informed of likely consequences of the changes. When poverty concentrations 
are considered, either to establish target arejs, to enable school-wide prdjects, or 
to allocate concentration grants to districts, the results of the uneverhnidonal- 
distribution of wealth can "be seen. The Southeast 'and South Central regions, aid 
large cities and rural areas, where poverty is more common, tend to be the • 
beneficiaries of policies that consider concentrations of poverty in addition to the 
considerations already in the Title' I allocation formulas and eligibility rules. 
■ \ fc 
Although the -reauthorization of Title I in 1978 did not contain major changes in the 
distribution formulas for funds, certain minor adjustments were made in an attempt to 
improve conformance of the program to its underlying intention to increase the equality of 
■educational opportunity. Amorfg these adjustments were clarifications about what have 
^ become known as 'concentration grants' and 'target areas.' 

Each year, .the Title I allocation is distributed to school districts by way of three types of 
grants/ Basic Grants are those sub-allocafrtews distributed" according to a complicated, but 
uniform procedure to all eligible districts. Because the imposition of a uniform procedure 
cannot meet all of the intentions of the Congress, two additional types of grants are also - 
made.* incentive Grants are\awarded to states that do better than'the national average in 
supporting compensatory education'. Through Incentive Grants, the federal government' 
encourages state and local education programs by rewarding those states that financially 
demonstrate their commitments to compensatory education. The third type of grant is the 
Concentration Grant, made tB districts with either large numbers or high proportions of 
poor children. These grants addresss the'belief that educational problems are compounded 
in areas with relatively high concentrations 6f poverty. The proportion of poor children 
that qualifies a district to receive a Concentration Grant^was set in 1978 at 20 percent (or 
5,000 students), but in the process of settings proportion it. was important to examine 
the results of various proportions in ferms of the numbers and types of districts that 
would qualify. t ^ • 

At the school level, the concentration of^poor children is almost important, because within^ * 
districts, school eligibility for Title I is determined by the concentration of poverty in the 
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school's attendance area. But even further among schools eligible under Basic Grants, 
those with certain concentrations of poverty can implement school-wide Title I projects in 
which all the students in the school can be served instead of only the lowest-achieving 
ones, when certain fiscal requirements are also met. When schools have very high propor : 
tions of students from low-income families (set in the Education Amendments of 1978 at 75 
percent), then after some local or state funds are added, all the children may be served by a 
school-wide program. In a similar vein, the concept of 'target area' establishes a uniform 
baseline Tor the automatic eligibility of all schools yvith a certain concentration of poor 
children (set at 20 percent in the Education Amendments of 1978). Implementation of this 
rule means thaf no school at or above the concentration level can be ineligible for Title I 
due to its low pdverty ranking within its district, white another school with a lower concen- 
tration is eligible because it ranked high in its (higlif r income) district. This can be seen as 
the imposition of an absolute standard onto the relative standards resulting from the 
distribution formula's sensitivity to the nation's complex economic and political structure. 

In summary, in terms of districts with concentrations of poor children, Concentration 
Grants would be awarded, while in terms gf schools with high concentrations, eligibility 
is assured and all students can be served. These exceptions to the usual allocations (via 
Basic Grants), are justified both on practical grounds (it's difficult not to serve'a small num- 
ber of students in a school), and on educational grounds (the high concentration of de- 
prived students is presumed to bave deleterious effects on the academic growth of all 
the students)^ . , 

Prior to and during the Congressional hearings 6n reauthorization, the U.S. Department of 
Education requested pertinent information from the SES regarding how' many districts, 
schools, and students would "be affected under implementation of the rules with different 
criteria for 'concentration.' The value of the SES data lay not onl^ in its high degree of- 
accuracy when statistically weighted foj the population (see Hoepfner, Zagorski, and- 
Wellisch, 1977), but also in its recency (collected in 1576 and 1977). The only alternative to 
these recent data Were dat&jfrom the 1970 census, .considered to be inadequate due to 
subsequent demographic changes. The information was to.be used in the following mag-^ 
ner. for various definitions of 'concentration/ we would estimate the numbers of districts, 
schools or students qualifying, and describe them in terms of regic)n.and urbanism. 

SCHOOLS IN TARGET *REAS ' 

In 1975-76, 4,750 principals in a nationally representative sample were queried about N the 
characteristics of their schools. In particular, they responded to one item that asked for an 
estimate pf the percentage of students meeting Title I poverty criteria (see the third chapter 
in this part) or the percentage of students eligible for free lunches. When the principal 
provided both percentages, the higher one was chosen as an estimate of the percentage of 
the school's enrollment from homes in poverty (the school's poverty concentration). The 
grade 1-6 enrollment, geographic location* and degree of urbanism were also obtained 
irom responses to the questionnaire. What follows are results of our tabulation for five 
definitions of 'concentration' counts. For each we present the numbers and types of 
schools qualifying. 



Schools With Ten Percent or^More Poverty 

We project that there are 47,146 schools offering any of the grades in the 1-6 range that 
, h^ve 10 percent or more of students in poverty. This amounts to 75.4 percent of the 
nation's elementary schools. Clearly then, if 10 percent were specified as the concentration 
of poor children ensuring school eligibility for Title I, then over 75 percent of the elemen- 
tary schools* would be automatically eligible. Given reasonable estimates of the future 
funding for Tit^e I, the implications are that either many schools would be defined as eligi- 
ble but not-be able to participate, or the program would have to be spreatj out to so many 
schools that the per-pupil allocation would be grossly insufficient for provision of programs 
' that are of adequate size, scope, and quality. 

Regional Distribution of the 47,146 Schools. Table 1-1 presents the projected distribu- 
tion of the '10 percent poverty' schools among ten regions. With the exceptions of the 
Southeast, North Midwest, and South Central regions the projected percentages of schools 
having 10 percent or more of poverty enrollments correspond closely to the percentages of 
elementary schools in the regions. A disproportionately greater number of schools have 10* 
percent or more poor children enrolled in the Southeast and South Central regions, 
disproportionately fewer schools in the North Midwest region. 

Table 1-1 

. Projected Regional Distribution of Elementary Schools 
With Poverty Concentrations at and Abpve 10 Percent 



T= * " ' , 

SchoolsAVith 10 Percent or More Poverty 


Region* 


Projected 
Number 


Projected 
Percentage 


Population Percentage 
of Elementary Schools 
With Grades 1-6 


New England % 
Metropolitan Northeast 
Mid-Atlantic 
Southeast 
North Midwest 


3,444 
3,351 
4,884 
8,668 * 
8,606 


7.3 
7.1 
10.4 
18.4 
18.3 


7.1 
8.3 
10.6 
14.6 
21.9 


South Central 
Central Midwest 
North Centra! 
Pacific Southwest 
Pacific Northwest 


6,161 
3,437 
v 2,257 
\ 4,354 
* 1,987 • 


13.1 
7.3 
4.8 
9.2 
4.2 


11.1 

7.1 ' 
5.2 
9.9 
4.2 


Total, 

4 


. 47,149** 


100.1** 


100.0 



•The regions are New^England : CN, MA, ME, NH^RI, VT, Metropolitan Northeast -~NJ, NY, Mid-Atlantic - DE, 
DC, MD, PA, VA, WV, Southeast - AL, FL, CA, KY', MS, NC, SC, TN, North Midwest - IL, IN, Ml, MN, OH, Wl; 
South Central - AR, LA, NM, OK, TX, Central Midwest - IA, KN, MO, NE, North Central -CO, MT,' ND, Sd! 
UT,WY; Pacific Southwest - A2, CA, HI, NV; Pacific "Northwest . AK, ID, OR, WA. 



** Discrepancies are due to rounding ercors 47,149 is 75.4 percent nationwide 
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Distribution of the 47,146 Schook by Urbjnibm. Table 1-2 presents the distribution of 
the '10 percent poverty' schools among seven categories of urbanism. Comparing the last 
two columns of Table 1-2, It can be seen that suburbs of cities have a smaller projected^ 
percentage of Schools with 10 percent or more poverty than its i>hare of schools in the 
population, while rural areas not near cities have larger projected percentages. Defining 
t$rg)et r areas iti terms of 10 percent concentrations of low-income children, therefore, offers 
potent*) benefits more to rural areas than to the suburbs. J&ffies would also experience 
some small benefits under this criterion. 



Table 1 : 2 

Projected Distribution of Elementary Schools With Poverty 
Concentrations at and Above 10 Percent, by Categories of Urbanism 





Schools With 10 Percerlt or More Poverty 




Urbanism 


Projected 
Number 


Projected 
Percentage 


Projected Population 
Percentage of Elementary 
Schools With Grades^1<6 


City or over 500,000 
City of 200,000-500,000 
City oi 50,00a 200,000 
Suburb ot city 
City under 50,000 
Rural area near city 
Rural area not near city 


1909 

3,033 

5,084 . 

4,285 / 
13,285 

4,969 
13.585 


8 3 
4 3 

28 2 . 
10.5 
28 8 


* 7 3 * 
3 9 
10.6 
14 8 

28.0 v * 

10.6 

24.9 


Total 


47,150; 


1000 


100 1 



•Discrepancies are due to rounding errors, 47,150 is 75 4 percent nationwide 



Schools With Twenty Percent or More Poverty 



Because of the large number of the nation's elementary scho&s that would be eligible for 
Title I with 10 percent concentrations of students from low-income families, similar tab r 
ulations were made witTi a 20 percent criterion. The national projection indicates that 
33.337 (53.3 percent) of the nation's elementary schools with grades in the 1-6 range have 
20 percent or more of their students from low-income' families. By. subtraction, using the 
figures from the first criterion above, we cari^ee that 13,809—21.1 percent— have poverty 
concentrations between 10 and 19 percent. Clearly, even this criterion allows more schools 
to be eligible to participate inT^tle I than are likely to be well served under reasonable 
expectations of future funding for the program. 

(• - 

Regional Distribution of the 33,337 Schools. Table 1-3 presents the distribution of the 
33,337 schools with 20 percent'concentrations of students in poverty across the ten regions 
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of the nation. Comparisons of the last two columns reveal that the Southeast and South 
Central regions have a highe'r proportion of schools with. 20 percent or more poverty 
enrollments than their population share of schools. Conversely the North Midwest region 
has^ew schools with 20 percent ortnore poverty enrollments irt comparison to its popula- 
tion share of schools. The distribution does not differ much from that found for schools 
with 10 percent or more poverty, but a noticeable difference is that the percentage 
becomes larger for the Southeast (18.4 percent in the 10 percenter more poverty case 
compared to 22.9 percent for the 20 percent case), while it decreases for the North 
Midwest (from 18.3 percent to 15.0 percent). The Southeast and South Central regions 
clearly have an advantage under the 20 percent criterion in terms of gaining/eligibility for 
schools, while the North Midwest and, to a lesser extent, the Central Midwest, have 
a disadvantage. 



Table 1-3 



• Projected Regional Distribution of Elemental Schools 
With Poverty Concentrations at and Above 2CWPercent 

* — 





Schools With 20 Percertt 




or More Poverty 




Region ■ 


Projected 
Number 




Projected 
Percentage 


Population Percentage 
of Elementary Schools 
With Grades 1-6 


New England 
Metropolitan Northeast 
Mid-Atlantic 
Southeast 
North Midwest 


2.243 
2,220 • 
3.541 
7,642 
4.992 




67 
67 
106 
22 9 
15.0 


- 7 1 
8.3 
10 6 
14.6 
21.9 


South Central 

Central Midwest 

North Central 

Pacific Southvyest 

Pacific Northwest * 


5,198 
1,922 
1,276 
.3,110 
1,196 




-1*5.6 ' 
58 
38 
,9 3 
.* 3,6 


11.1' 

IF 

7.1 

5 2 

9 9 , U*S 
4.2 


Total 


33,340* 




/1000 


. 100.0 



*Oiscrepanciesare due to rounding errors, 33,340.is 53 3 percent nationwide 



Distribution of the 33,337 Schools^ by Urbanism. "the projected distribution of the 
, 33,337 schools with 20 percent or more poverty enrollments over the^even categories of »► 

uibanism*f5*presenfed in Table 1-4. By comparing the last. two columns, it i^s-seen thatlarge i- 

cities and rural areas not near cities have greater percentages of schools with 20 percent or 
^ipore poverty enrollments than their population percentages. In contrast, suburbs have 
'considerably smaller percentages than their population percentages of elementary .schools. 

Not surprisingly,-»the .distribution is quite similar to that of schools with 10 percent or 

more poverty. ' 




■ \ 



Table 1-4 

Projected Distribution of Elementary Schools With Poverty 
Concentrations at and Above 20 Percent, by Categories of Urbanism 





Schools With 20 Percent or More Poverty 




Urbanism 


Projected 
Number 


Projected 
Percentage 


Projected Population 
Percentage of Elementary^ 
Schools With Grades 1-6 


City of over 500,000 
City of 200,000-500,000' 
City of 50,000-200,000 
Suburb of city 
City under 50,000 
Rural area near city 
Rural area not near city 


3,549 
1,627 
3,690 
2,143 
9,112 
2,950 
10,268 


10.6 
4.9 

11.1 
6.4 

27.3 
8.8 

30.8 


7.3* 

.10.6 ^ 

14.8 t 

10.6 / 2J 

24.9 ( 


Total 


33,339* 


' . 99.9 


100.1 



•Discrepancies are due to rounding errors; 33,339 is 53.3 percent nationwide. 



Schools With Thirty Percent* or More Poverty 

'Our projections estimate that 22,973 (36.7 percent) of the nation's elementary schools with 
grades in the 1-6 range have 30 percent ol more of their enrollments from poverty 
backgrounds*. We can infer from these and earlier figures that 10,364 schools— 16.6 percent 

'of the nation's elementary schools— have between 20 to 29 percent poverty enrollments. 

Regional Distribution of the 22,973 Schools. Inspection of the last two columns of 
Table 1-5 indicates that the Southeast and South Central regions have greater percentages 
of schools with 30 percent or more poverty enrollments in comparison with their popula- 
tion percentages. The table also reveals that the 'percentage of schools with 30 percent or 
more poverty enrollments in the North Midwestjs just slightly more than half of its popula- 
tion of elementary schools. Again, we can observe the tendency of an increasing share 
(from 22.9 percent in the case of 20 percent or more poverty, to 27.9 percent here) of 
schools meeting the poverty criterion in the Southeast region as the cutoff percentage of 
poverty enrollments is raised. On the other hand, there is a decreasing share (from 15.0 
percent in the case of the 20 percent cutoffto 12.6 percent when the cutoff is 30 percent) in 
the North MifJwest region. 

Distribution of the 22,973 Schools -by Urbanism. The last two columns of Table 1-6 
reveal that the percentages pf schools with 30 percent or more poverty enrpllments are 
larger for large cities (over 500,000) and rural areas not near cities in comparison to their 
population percentages of elementary schools with grades in the 1-6 range. As one raises 
the school poverty criterion "from M0 percent to 20 percent to 30<percent, large cities gain in 
'their shares of schools in target areas. In contrast, the share of schools meeting the poverty 
criterion in suburbs decreases from 9.1 percent to 6.4 percent to 4.7 percent. 



Table 1-5 

■5 

Projected Regional Distribution of Elementary Schools 
With Poverty Concentrations at and Above 30 Percent 



Schools With 30 Percent or More Poverty 


Region 


Projected 
Number 


Projected 
. Percentage 


Population Percentage 
of Elementary School? 
With Grades 1-6 


' New England 
Metropolitan Northeast- 
Mid-Atlantic 
Southeast 
North Midwest 


,1,450 
1,517 
. 2,503 
6,409 
2,902 


6.3 
"6.6 
10.9 . 
27.9 
12.6 


7.1 
8.3 
10.6 
14.6 
21.9 


South Central' 
Central Midwest 
North Central 
Pacific Southwest 
Pacific Northwest 


4,002 
997 
639 

2,009 
547 


17.4 
4.3 
>2.8 

' 8.7 \ 
2.4 


11.1 
7.1 

. 5.2 
9.9 

*T.Z 


Total 


22,975* 


99^9 


, 1 s 

1P0.0 


r 

* Discrepancies are due to rounding errors; 22,975 is 36.7 percent nationwide. 
* 
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Table 1-6 




Projected Distribution of Elementary Schools With Pbverty 
* Concentrations at and Above 30 Percent, by Categories of Urbanism 


Schools With 30 Percent or More Poverty , 


Urbanism 


Projected 
Number 


Projected 
Percentage 


Projected Population 
Percentage of Elementary 
Schools With Grades 1-6 


City of over 500,000 
City of 200,000-500,000 ' 
City of 50,000-200,000 
Suburb of city 
City under 50,000 
Rural area near city 
Rural area nc»t near city 


3,097 
1,244 
2,836 , 
1,076 
5,858, 
1,676 
•7,188 


13.5 
5.4 

* . 12.3 
4.7 
25.5 
7.3 
, 31.3 


' 7.3 t 
3.9 
10.6 
14.8 
28.0 
10.6 
24.9 


Total 


22,975* 


100.0 


"100.1 


•Discrepancies are due to rounding errors; 22,975 is 36.7 percent nationwide. 



Comparisons Among Poverty Percents 

Figures 1-1 and 1-2 present the data from Tables 1-1, 1-3 and 1-5 and from Tables 1-2, 1-4, 
and U6 t respectively, in graphic form for ease of seeing tKe effects by region and urbanism 
of the various criteria for, target areas. With higher and higher criteria, the Southeatt and 
South Central regions and large cities and rural areas gain i,n the percentages of eligible 
schools. At the same time, suburbs and the North Midwest lose. . % \. 

Schools With Poverty Enrollments of 50 or More Students 

■9 

So far, we have limited our concern with poverty concentrations to criteria expressed a$ 
percentages of school enrollment. Percentages based on small enrollments may have 
different educational implications from those based on large enrollments, so we did similar 
tabulations using an absolute criterion of 50 or more low-income students. Use of an 
absolute number like 50 provides a less relative assessment of the potential impact of 
poverty, and would be expected to reduce the rate of eligibility in sparsely ^populated areas 
that have small schools. We project that ^1,743/or 50.8 percent of the elementary schools 
with grades in the 1-6 range have 50 or more low-income students. 

Regional Distribution of the 31,742 Schools, inspection of Table 1-7 repeals that the 
Southeast has the largest number of schools with poverty enrollments of 50 or, more 
students; the North Midwek has the smallest number of such students. 

Table \-7 



Projected Regional Distribution of Elementary Schools 
With 50 or More Low-lhcome Students 



Schools With 50 or \iore Poor Students 


Region 


Projected 
Number 


Projected 
Percentage 


Population Percentage 
of Elementary Schools 
With Grades 1-6 


New England 
Metropolitan Northeast, 
Mid-Atlantic 
Southeast 
North Midwest 


1,688 
' 2,714 
3,521 

7,915' - 
4,928 


* * 5,3 
8.5 
1 1.1 
24.9 
15.5 


r-t ; 

7.1 
8.3 

10:6 

14.6 

21.9' . 


South Central 
Central Midwest 
North Central 
Pacific Southwest 
"Pacific Northwest* 


4,636 
' * 1,266 
815 
3,254 

1,008 

* * * 


14 6 
9 . 4.0 
2.6 

10.3 * 


V ■ n, • 

' 7.1 

° 5.2 
/ 9,9> 


Total f 


31,745* 


• 100.0 


100.0 



* Discrepancies are due-to rounding errors: 31,745 is 50 8 percent nationwide. 
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^ Figure 1*1 

Percentages of the Nation's Elementary School Students in Each Region 
Attending Schools Eligible Under Each of Three Criteria 
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Figure 1-2 . 

Percentages of the Nation's Elementary School Students, by Urbanism, 
Attending Schools Eligible Under Each of Three Criteria. 
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Distribution of the 31,743 Schools by Urbanism. Table 1-8 shows that urban areas (the 
first three categories within urbanism) have the largest numbers of schools with 50 or more 
-students in\poverty. Rural areas do not suffer a large disadvantage with this criterion, but 
the suburbs do. 

Table 1-8 

Projected Distribution of Elementary Schools With 50 
orfiore Low-Income Students, by Categories of Urbanism 



t Zr ^ ^ ^ ^ — 

Schools With 50 or More Poor Students 


^Urbanism 


Projected 
Number 


Projected 
Percentage ? 


Projected Population 
Percentage of Elementary 
Schools With Grades 1-6 


City of over 500,000 * 
City of 200,000-500,000 
City of 50,000-200,000 
Suburb of city 
J^ify under 50,000 
Rural area near city 
Rural area not near city 


3,769 
1,734 
; 4,050 
3,142 
9,078 
2,857 ' 
7,115 


11.9' 
* 5.5 
12.8 
9.9 

28.6 • 
9.0 
22.4 


7.3 

3.9 
10.6 
14.B 
28.0 
10.6 * 
24.9 


Total 


31,745* 


100.1 


100.1 



* Discrepancies are due to rounding erroq; 31,745 is 50.8 percent nationwide/ 

Schools With 10 Percent or More Poverty or 50 or More Low : lncome Students 

finally, we present the number of schools in the nation with either 10 percent or more 
poverty or 50 or m<?re students from low : income families. We estimate in Table* 1-9 
that 47,397 (75.8 percent) of the nation's elementary schools with grades in the 1 -grange 
have either 50 or more students in poverty or 10 percent or more of their 1-6 enrollments in. 
poverty. Note that the use of the 10 percent criterion alone yielded 75.4* percent; so the 
addition of another possibility— 50 or more students— did not add many schools. More than 
half of the schools me^t both of the criteria if they meet one. * 

Table 1-9 * 



Projected Numbers and Percentages of Elementary Schools 
With Either 10 Percent or More Poverty 
or 50 or More Low-Income Students 





9% or Less Poverjy 


10% or More Poverty 


Tofc*l 


49 or fewer 
low-income students 


l'5,130 • 
24.2% 


15,655 
25.0% 


. 30,785* 
49.2% 


50 or, more 
low-income students 


251 
0.4% 


» 

31,491, 
50.4% ^ 


31,742* 
.50.8% . 


Total 


15,381 
24.6% 


47,146 
75.4% 


62,527 
100.0%* 



•Discrepancies are due to rounding errors 



DISTRICTS ELIGIBLE FOR CONCENTRATION GRANTS * v 

The discussion so far has focused on the characteristics of schools for determining their 
eligibility to offer Title I services. The concept of poverty concentrations is also relevant to 
districts for determining their eligibility to receive a Title I Concentration Grant. We 
therefore used the SES representative sample of districts to project eligibility rates for 
Concentration Grants. f 

Table 1-10 presents national estimates by tenths of district-poverty concentrations. The 
projections are based on 219 superintendents' estimates of the percentages of students in 
their districts that met the Title I poverty criteria during the 1976-77 school year. This 
sample was/projected to 14,683 districts that served children in the grade 1-6 range. 

If additional funds were allocated to districts with 20 percent or more enrollments of 
children from low-income families according to these 1976-77 data, we estimate that 48 
percent of the nation's districts* would qualify for them. About one-fourth of th§ nation's 
students are enrolled in schools in thpse districts. 



Table 1-10 



$ 



Projected Distribution, of the Nation's School Districts, 



by Percentage of Students in 


Poverty 


Percentage of 


National Projections From 219 Districts in the SE? 


*"* * Students in poverty ' 


Number of Districts 


Percentage of Districts 


90-100 ~ 




'0.8 


80-89* 


15 


0 1 ' 


70-79 




2.1 


, 60-69 


. 136 


0.9 


50-59 ^ 


928 ' 


6.3 


4 

J 40-49 


1,695 


11.5 


\ 30-34 ' 


1.023 


7.0 


20^9 


2,828 


19.3 


'10-19 


5,178 


35.3 


a - 0-9 ' 


2.462 ' 


16.6* . 


« Total' 


14,683 


100.1 







•It is- interesting to note the data used fotfdeterminmg district eligibility for Concentration Grants under Sec- 
tion 1 17 of Title I result in about half the nation's districts receiving such grants. About 80 percent of them 
-are in urban settings • * * 
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SCHOOLS ELIGIBLE FOR SCHOOL-WIDE PROJECTS 

At i certain saturation of a school's enrollment with low-income Children, it seems prac- 
tically and educationally unnecessary to go through the trouble of assessing student need 
and then selecting among them for Title I services. The chances are presumed high in tRese 
cases that-all students are in need and the school's Title I resources could be (beneficially)- 
distributed^ all ot them. This is the rationale behind'the 1977-78 discussion about school- 
wide projects for the Education Amendments. To inform this discussion, we were asked to 
provide the Office of Education with national projections of schools with ,70 percent and 80 
percent enrollments of low-income children. 

Schools With 70 Percent or More Poverty. We estimate that 9.2 "percent (5,724) of the 
nation's schools with grades in the 1-6 range have enrollments of 70 percent' or more of 
children from low-income families. These schools serve an estimated 2,150,279 students 
(about 10.3 percent of the nation's public school students enrolled in grades 1-6). The dif- 
ference in the school and student percentages indicates that the high-poverty schools tend 
to be larger than average. 

Tables and 1-12 present the distribution of these schools by region and by urbanism 
From the tables^we can see that the Southeast and South Central regions have an abun- 
dance of these high-poverty schools, wNJe the North Midwest has relatively few of them 
Similarly, large cities could have more/ school-wide projects under this criterion and 
.suburbs and small cities could have few^t of thenv — 

* 

Table 1-11 



Projected Regional Distribution of Elementary Schools 
With Poverty Concentrations at and Above 70 Percent 





Schools With 70 Percent 


or More Poverty 




Region 


* Projected 
Number 


Projected 
Percentage 


Population Percentage 
of Elementary Schools 
With Grades 1-6 


New England 
Metropolitan Northeast 
Mid-Atlantic 
Southeast 
North Midwest 


425 
457 
513 
1,620 
590 


7.4 

8.0 $ 

9.0 
28.3 
10.3 


7.1 ' 
8.3 
10.6 
14.6 
; 21.9 


South Central 
Central Midwest 
North Central 
4 Pacific Southwest 
Pacific "Northwest 


1,172 
,176 
124 
528 
120 


20.5 
3.1 
2.2 
9.2 
2.1 


11.1 
7.1 

5.2 - 

9.9 

4.2 


Total 


,5,725* 


100.1 


100.0 



•Discrepancies are due to rounding errors; 5,725 is 9.2 percent nationwide. 



Table 1-12 



Projected Distribution of Elementary Schools With Poverty 
Concentrations at and Above 70 Percent, by Categories of Urbanism 



I — J 

Schools With 70 Percent orMofe Poverty 


Urbanism 


Projected 
Number 


* 

Projected w 
Percentage 


Projected Population 
Percentage of Elementary 
Schools With Grades 1-6 


City of over 500,000 
City of'200, 000-500,000 
City of 50,000-200,000 
Suburb of city 
City under 50,000 
Rural area near city 
Rural area not near city 


1,674 
426 
633 
88 

' 1,019 * 
208 
1,*676 


29.2 
. *4 

im 

h5 
17.8 

3.6 
29,3 


7.3 
3.9 
10.6 

14.8 : * 

28.0 

10.6* 

24.9 


Total 


5,724* 


99.9 n 


100.1, 



•Discrepancies are due to rounding errors; 5,724 is^^ percent nationwide. 



Schools with 80 Percent or More Poverty. When the criterion for school-wide projects 
is 80 percent, the percentage of qualifying schools is 5.8 (3,638 scKbols). We-project that 
1383,008 students- (abbut 6.6 percent) are enrolled in the schools with 80 percent or more 
of students from low-income families. Tables t-13 and 1-14 present the distributions of 
these high-poverty schools over region and urbanism. „ 



Table 1-131 



projected Regional Distribution of Elementary Schools 
QWith* Poverty; Concentrations at and Above 80 Percent 



Schools With 80 Percent or More Poverty 



Region 



Projected 
Number 




jected 
ercentage 



Population Percentage 
of Elementary Schools 
With Grades 1-6 



New England 
Metropolitan Northeast 
Mid-Atlantic 
Southeast 
North Midwest 

Soutn CentraJ 
Central Midwest 
North Central 
Pacific Southwest* 
Pacific Northwest 



192 
329 
233 
1,033 
442 

785 
105 
102 

318 - 
98 




7.1 
8:3 
10.6 
14.6 
21.9 

11.1 

Z.v 

5.2 
9.9 
4.2 



Total 



3,637* 



100.0 



100.0 



•Discrepancies are due to rounding errors; 3,637 schools is 5.8 percent nationwide. 



Table 1-14 



Projected Distribution of Elementary Schools With Poverty - * 
. Concentrations at and Above 80 Percent, by Categories of Urbanism 





7* 

Schools With 80 Percent or More Poverty 




Urbanism 


Projected 
Number 


Projected 
Percentage 


Projected Population 
Percentage of Elementary 

Qs-hrvtlc With f^w-xAtzc t C 

x.noois wiin vjraaes l-o 


City of over 500,000 
City of 200,000-500,000 
City of 50,000-200,000 
Suburb of city 
City under 50,000 
Rural area near city 
Rural area not nearxity 


4 1,282 
•276 
367 
43 
510 
75 
1.086 


" 35.2 
7.6 

10,1 
1.2 

14.0 
2.1 

* ' 29.8 


3.9 
10.6 
14.8 
28.0 v 
10.6 
24.9 


Total 


3,639* 


100,0 y 


/ 

. J00.1 



•Discrepancies are due to rounding errors; 3,639 schools is 5.8 percent nationwide. 



Based in part on. these projections, an intermediate level (75 percent) was specified in the 
amendment law. 

In summary, when the concentration of children from low-incorqg families is considered,, 
either to ^teblish target areas, to enable school-wide projects, or to .allocate concentration 
grants to districts, the results of the uneven national distribution of weaJth can be seen. The 
Southeast and South Central regions and J^rge cities and rural areas, where poverty is^more 
common, tend to be the beneficiaries of policies that consider concentrations of poverty in 
addition* to the consideration already in the Title I allocation formula^ 
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CHAPTER 2. ESTIMATES OF SCHOOL POVERTY 



c 4 

Joel M. Moskowitz and Deborah S. Brown 

Districts are allowed to use a variety of ways to estimate the numberW propor- 
tion of poor Children in schools for the purpose of determining the eligibility of 

. schools to offer Title I services. Jt is important, therefore, to know how Ihe 
different ways result in estimates of different accuracy. Five estimates of the con r 
centration of poor students in schools were developed and tested. Although the 
estimates are moderately to higtyy intercorrelated, they 1 result in significantly 
different average estimates. Using the Federal Allocation Formula applied to in- 
come data obtained from home interviews as the standard and accurate measure 
of school poverty, estimate^ by principals we're found to be significantly and con- « 
sistently lower; imd estimates by teachers and by application of district allocation ' 
formulas to be very close tqthe standard. The most commonly Used estimates of 
school poverty levels, principals' reports, while resulting in significantly lower 
absolute estimates, are so consistent in their underestimation that districts' 

* ordering of schools can be expected to conform well to the Title I intentions. 

Th 4 e -distribution of Title I money from the federal level through the states, counties, 
districts, and schools-ending up as additional -services for specific kinds of students— is a 
complex procedure. In essence, the flow is as follows: 

• A federal formula, based largely on income, allocates funds to the counties, through 
the states. The^frmula counts the number of children aged 5 to 17 from families 
below the Orshansky poverty level, children aged 5 to 17 from families receiving 

- payments under AFDC of a specified ^mount, and children aged 5 to 17 supported by 
public funds. These counts come from census dataj but are generally available only 
down to the county le^el. . / 

« 

• At the county level a state formula allocates funds to districts (where districts and 
counties are coterminous, this step is ignored). The state formula may use a different* 
definition of poverty from that in the federal formula, but the definition must meet 
Title I statutes. In essence, the formula requires data that the< state considers to best 
reflect the current and accurate distribution of eligible children from low-income 

-families. In many stales these data are very similar to those used in the federal formula, 
but several states use only AFDC counts, free-lunch counts, Ideal survey, old census 
counts, or local knowledge of poverty concentrations. 

• The districts, in turn, must determine the schools in which Title I services are to be n 
provided and the level of services allocated to each school. The district .must identify 
eligible schools, based for the most part on counts of poverty children in the atten- 
dance area, and then select the schools that will receive the services. The district can 
count low-income children, by attendance area, in any of the ways available to the 
states, or by .school surveys, or health, housing or, employment statistics. The method 
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employed may be a combination of any or all of these counting methods, but must be 
applied to alf schools consrstently. Having ranked the schools, each district must 
select those to receive the. Title I services through another set of procedures that 
generally prohibit the 'skipping' of one school in need for another of lesser need, and 
that requires the Title I program to be of sufficient size, scope, and quality to give 
reasonable promise of success in meeting the goals of the program. 

After target sdiools are identified from those eligible, the district must determine what 
kinds of students to serve (grades or educational ^problems), how many to serve of 
*each kind, and the amount of funds to be allocated to each student's services. The 
only requirement is that children* most in need of special assistance are to be served. 

It is at the point where districts determine school eligibility that we focus our attention. 
Districts generally use 1970/census counts, current AFDC counts, and/or counts of free- 
lunch recipients^whichever are available— as the indicators of poverty concentration for 
schools. An NIE survey (NIE, 1977) collected counts of poor children in all schools in TOO 
Title I districts in 1975-76 according to three measures. 1970 census, enrolled AFDC 
children, and free-lunch recipients. The numbers of children identified by the three 
^methpds Were correlated to see how similar they were. The correlation between census, 
counts and AFDC children was + .58, between census and free lunch, +.51; and between 
"AFDC affiMree lunch, + .81 . The AFDC and free-lunch counts are th^ most similar, but the 
accuracy of any single method remained undetermined. . . ^ 

In the present study, we attempted to obtain accurate data on poverty concentrations of 
schools, and then to compare these data with the kinds of data typically used in practice, 
i.e., data in the form of principals' reports. As part of the Sustaining Ejects Study (SES), prin- 
xipals were asked to estimate the percentages of poor students in their schools and teachers 
judged the poverty status of their students. We then examined how well these estimates re- 
lated to accurate information derived from home interviews with parents. We also checked 
to see if certain characteristics of schools and principals influence these relationships. 

T^t ESTIMATES OF SCHOOL POVERTY 

Information about the concentration of poor students was obtained for 242 nationally 
representative schools. The data were obtained from home interviews of parent^ and from 
judgments made by principals and teachers. 

The Accurate Estimates. One of the mandates of the SES was to obtain accurate infor- 
mation on family economic status for a representative subsample of students. Between 15 
and 200 home interviews were completed with parents children participating in each 
school of the representative sample. Data from the 45,579Lcompleted interviews could be 
projected to population estimates because of the methods used in constructing the sample. 
Income data from the interviews, fully described by Breglio, Hinckley, and Beal 
(1978) could als6 be transformed into standard indexes of poverty, ^d the percentage of 
*poor students in each school could serve as an unbiased and accurate estimate of the 
school's poverty. 

*■ > * 
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TtKMndex of poverty is referred, to as 'current low income,' which is the percentage C 
students in each school meeting the criteria used by the federal government to allocate 
Title I funds to counties ('Federal Allocation Formula'). As noted above, these criteria are 
based on the number of families receiving AFDC, the number below the Orshansky poverty 
level, and the number of students residing in foster homes and institutions for the neglected - 
or delinquent. Based on their national projections, Breglio et al. estimated 4.2 million such/ 
students, or 20.9 percent of the population of elementary school students: 

Estimates by Principals. During c the previous year (the 1975-76 school year), i princi- 
pals were asked to report poverty estimates for their schools. They responded to the follow- 
ing request: • v 

. , .* i 

To estimate the precentage of students in your school who meet ESEA Title I ^poverty 
criteria, please answer (a) or (b), whichever is easier for you to estimate. 

(a) Pecentage of students in your school who meeHhe'distnct's ESEA Titl<* I 
poverty criteria j %. 

(b) Percentage of students in your school who are eligible to "receive free 
lunches %. m / 

Thus, the principals estimated either the percentage of students meeting district Title I 
school poverty criteria ('Principal Report of Title I Eligible') or the percentage eligible for 
free luncN'Pnncipal Report of Free-Lunch' Eligible'). Eighty-two percent of the principals 
chose to estimate the percentage eligible to '-receive free lunch, whereas 47 percent' 
estimated the percent of Title I eligible. (Thirty-one percent ,of the principal? reported 
both estimates.) ^ j 

Estimates by Teachers. Each homeroom teacher in each school estimated the poverty 
status of each of his/her students. The teachers' estimates were made m response to the 
following item from the Student Background Checklist: / 

■J 

To your knowledge, does the student participate^ in a free or reduced-priced lunch' 
or breakfast program? (Use your own observation or school* records. Do not ask 
, the student.) 

• Yes • No / v 

I -so • 

Responses were aggregated within schools to obtain estimates of school, poverty that we 
call 'Teacher Report of Free-Lunch Participation.' 



Estimates at the District Level. The criteria used to allocate Title I services to sc 
vary by district. Most districts employ multiple criteria, with the average district usin 
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The three most frequently used criteria ^Wang, Hoepfner, Zagorski, Hemenway, Brown/ 
and Bear, 1978) are/ in order of frequency: 

1. Free or reduced-price lunch count,,* ^ „ ' 

2. AFDC enrollment, and % ^ 

3. Qensus data on family income. • > * w " 

" Because each district indicated which criteria were used to allocate Titte I services*tp 
, . schools, and because we. had independent information on each criterion, we could 
' aggregate all relevant 'criteria to the school , level and estimate for each school the 

percentage of students meeting the cfistrict'si Title I school-eligibility criterion ('District 

Allocation Formula'). v * 7M, 

Among the various estimates of poverty* we would expect the 'Principal ^Report of Free- 
Lunch Eligible' to be highly related to the Teacher Report of Free-Lunch Participation,' 
since the percentage of students in each school eligible to receive free lunches, as 
estimated by the princip^J, should be comparable to the percentage participating in the 
lunch or breakfast programs as reported by the teachers, albeit a year later. The principal's 
estimate should also be .highly related to the 'District Allocation Formula,' as most districts 
use free^lunch count* as a criterion* The 'Principal Report of Title I Eligible' should be 
related to the 'Djstrict Allocation Formula,' because, aS noted above, principals ,were 
directed to make "this specific poverty estimate. This principal's estimate should also be 
strongly related to the two poverty estimates based on free lunches because they share 
severaf common aspects in their definitions. 

, Additional "analyses will determine whether specific school er principal characteristics in- 
fluence the relationships between the various poverty estimates. Specifically, we consider 
the following characteristics that might be suspected to reduce acaAacy!^ 

• Stability of the student body / 

• Type of compensatoryreducation funding received 

• Urbanism of school 

• * Minority concentration of school " 

• Percentage of students bused for racial balance ' " 

• Enrollmerjt of school 

• . Principal's tenure in the school 



RELATIONS AMONG THE POVERTY ESTIMATES 

. 

As may be seen in Table 2-1, principals' and teachers' estimates of students .who are eligible 
for or participating in free-lunch programs (C and D, respectively) are most highly related to 
the poverty index we believe is most accurate;, namely our data collected through home 
interviews and used in the Federal Allocation. Formula (A). Principals' reports pf students 
eligible for Title I (B) and the district formulas (E) are slightly less correlated with our accu* 
rate measure. Looking at it another way, the principals' estimates of free-lunch eligibility 
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has very high concurrent validity with all the other estimates, and correlated highest with 
the most accurate measure, that obtained from the'home interviews. . ^ 

'Principal's' .estimates of the percentages of students meeting district Title I eligibility criteria 
(B) are only moderately related to Ihe percentages of students actually meeting'these 
criteria (E) one year later (r - .54). These principal estimates were more highly related to 
the. percentages of students meeting Federal Allocation Formula criteria (r = .69), and to 
the percentages of students participating in free or reduced-priced lu/ich or breakfast pro- 
grams one year later (r = .78). .Hence, the principal's Title I eligibility estimate is a much 
fetter indicator of free-lunch participation than it is of the percentage of students meeting 
the criteria established by the district for the school's Title I eligibility? This is not surprising 
as the latter^stimate is complex and generally includes several different criteria, with free- 
lunch participation as only one of its components. What is somewhat surprising. is that the 
« principal's free-lunch estimate (C) is a better predictor of the school's percentage of district 
Title I eligibles (E). 

Not surprisingly, the free-lunch eligibility estimate is a better indicator of free-lunch par- 
ticipation. Hence, the principal's free-lunch estimate is superior to the principal's Title I 
estimate on several counts. * 

The means and standard deviations of the various poverty estimates, computed from all the 
casqs available for each estimate, are presented in Table 2-2. We can see that home 
interviews (A) provide the highest average estimate of school poverty with district formulas 
being very similar. Principals' reports of Title I eligibles result in the lowest mean estimate 
Comparing these means^to the correlations in Table 2-1, we can see that, although 
principals' estimates of free-lunch eligibles almost highly correlated with the Federal 
Allocation Formula, they are considerably lower. (Correlation coefficients, of course, are 
insensitive to differences in the overall levels of the variables being correlated.) Of course, 
this consistent difference is not crffeal in cases where the estimate is applied uniformly 
across^a set of schools, as when ranking tjiem for Title I eligibility. 



Table 2-1 



Intercorrelatlons Among Poverty Estimates* 



Poverty Estimate 



A 


B 


C 


D 




E 


113 


194 


238 


238 






.69 




73 


113 


113 


.76 


.79 




" 194 




194 


v .75 - 


.78 


.88 ' 


t 




238 


.68 


.54 


.77 


.74 



















A. Federal Allocation Formula**" 

B. Principal Report of Title I Eligible 

C Principal Report of Free-Lunch Eligible 

D. Teacher Report of Free-Lunch Participation 

E. District^ Allocation Formula 



cT^I Y w,vvv : MC numu^ or scnoois commuting to their calculation above. 

S> ol t T^IJ 5 ? P r,na ^ c . hose to provide different estimates, and because some districts did not 

Dart Cl Date n Tif P anH haH nn -a m-->t.An ^,r»v.,U , 



. participate in Title I and had no allocation formula. 

**This jstimate, based on home interviews, is considered to be the most accurate. 
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Table 2-2 

Means and Standard Deviations of Poverty Estimates 



Poverty Estimate 


N ' 


Mean 


S.D. 


a 

Federal Allocation Formula <A) 


238 


39 9 


19.1 


Principal Report of Title 1 Eligible (B) * 


- 113 


28.8 v 


^ 23.8 


Priqcipal Report of Free- Lunch Eligible (C) 


194 


33.5 


^ 25.8' 


Teacher ReporXof Free-Lunch Participation (D) 


238 1 


38.5 


' 28.7 


District Allocation Formula (E) 
1 1 — 1 — 


238 


' 39.8 


1 25.9 



Comparing selected means with a^means with a Mest for paired observations, we find 
that the principal underestimates the percentage of Title I poverty eligibles, as compared 
to the percentage eligible by district criteria (t (112) - 4.65, p<.'001). The principal also 
underestimates the percentage of free-lunch eligibles as compared to the percentage 
receiving free or reduced-price lunch or breakfast (the number eligible should, of course, 
exceed the number participating), as reported by the teachers one year later (^93) - 4.78, 
p<w(X)1). The generality of statistically significant differences among the mean estimates 
conffrms intuitive knowledge that none of the poverty estimates is perfectly comparable to 
another poverty estimate. • -1 

THE EFFECTS OF SCHOOL AND PRINCIPAL CHARACTERISTICS ON THE ESTIMATES 

The relationship between the principals' estimates of free-lunch eligibility (C)'and the 
teachers'- estimates 6f free-lunch participation (D), vyas exaVnined through' cross-tabulation, 
controlling for various school and principal characteristics as third variables. Similar 
analyses were performed on th$ principals' Title I eligibility estimates and the measures p(. 
Title I eligibility ba^epl on district criteria. ' 

«* * " * 

The Results -are reported Jn Table 2-3. .Gamma is a measpre^f association between two 

ordinal-level variables, yielding values not unlike those of a product-morYient correlation 
coefficient*. Zero-order gamma measures the relationship between two variables, e.g., 
Principal Title I Eligible estimate of poverty and District Allocation Formula estimate, 
Without controlliryg for any other variables. First-order partial %amyna measures the 
-relationship between the twp variables controlling for one qthe^ variable, e.g., Student 
Body Stability. By comparing the zero-order gamma with the partial gamma, one 
can deterfttine^whqther the third variable moderates the relationship between the other 
two. As can be seen in the table, in no instances were s'chool or principal characteristics 
found to mocterate the relationships between the poverty estimates. This finding presents 
indirect evidence that the characteristics do not influence the levels of the poverty 
estimates, either. However, .if the characteristics influenced the^ levels of both poverty 
estimates similarly (which we don't consider very likely), no matter how strong.the in- 
fluence, the reduction in the partial gamma would be very small. 
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Table 2-3 



The Effects of Controlling for School and Principal Characteristics 
/ on the Relationships Between* Two Title I Estimates and • 
Between Two Free-Lunch Estimates 



Characteristic > 
Controlled for 


Principal Title I Eligible 
and 

District Allocation Formula 


■m 

Principal Free-Lunch Eligible 
and 

* Teacher Free-Lunch Participation 




First-OrHpr Partial C.xmmx 
■ list viuci ralllal KjgMllilHa 


j First-Order Partial Gamma 


(Zero-Order Gamma) 


(.66) 


(.91) 


Student body stability 
CE funding 
Urbanism of school 
Minority concentration 
Busing for racial balance 
School size (enrollment) 
^Principal's tenure • * 


9 <i 

.62 

" ^ .62 
.60 
.58 
.66 . 
.65 

< . .66 


.91 ' 
91 

' 91 
.88 
' .90 
> * * .9t n \ 

s .91 



CONCLUSIONS . 

The estimate^bf school poverty are almost all highly related to one another, but they result 
in different levels of concentration of poverty. Using the Federal Allocation Formula 
applied to income data obtained from home -interviews as the standard and accurate 
measure of school' poverty, estimates by principals are significantly and consistently lower; 
estimates By teachers and by application*- of district allocation formulas are very close to 
the-standard. • . 

Principals 4 estimates of the percentages of students eligible for free-lunch programs are 
^superior to their estimates of Title I eligibility for two reasons. First, the free-luncrvestimate 
. is a better indicator of the percentage of poor students in the school when poverty is 
defined by either district or federal allocation criteria. In addition, principals generally 
^ prefer to estimate the percentages of free-lunch eligibles. 



CHAPTER 3: CRITERIA FOR DETERMINING TITLE I PARTICIPATION 

OF SCHOOLS ' 



Ralph Hoepfrler 
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Information on how districts determine which schools will participate in Title I 
are projected to the nation and contrasted with previous findings. 



After federal Title I funds are allocated to counties on the basis of the number of children 
aged 5 to 17 from low-income families, those receiving welfare payments, and those sup- 
ported by public funds, and the states have allocated the funds to school districts on the 
basis of some indication of the poverty level of the district's attendance area, the districts 
must select the schools to participate in the Title I program (see Moskowitz and Brown, this 
volume). Districts are to identify eligible .schools on the basis of concentrations of poverty 
children in the schools' attendance areas. The methods used to obtain the poverty counts 
are not restrictive, so that districts can use a wide variety of more orjess effective methods 
■ for ranking schools, so long as they provide current, accurate estimates of the numbers of 
eligible children from low-income families. Among those methods, districts can use census 
data, AFDC counts, free-lunch counts, free-breakfast counts, local surveys, old census 
counts, local knowjedge about poverty concentrations, school surveys*or health, housing, 
or unemployment statistics. 



Implications from the NIE Compensatory Education Study (NIE, 1977b) are that some of the 
more obscure methods are rarely used. Based on their'natiqnally representative sample of 
TOO TitleJ districts, but not^rojected to population estimates, NIE reported that 67 percent 
used 1970 census data family income; 66 percent used /fee-lunch and/or free-breakfast 
counts; 51 percent used counts of AFDCchildren; and 34 percent used other economic 
data, such' as housing ahd unemployment statistics or local surveys. The numbers add up to 
more than 100 percent because many districts reported using more than one source 
of poverty data-. (The Demonstration Study of- NIE, 1977a, showed that, when given 
th6 opportunity, districts .interested in trying something new usually abandoned the 
poverty-based procedures for selecting schools and adopted eligibility criteria based 
on achievement.) , 

c 

The 219 districts-in the Sustaining Effects Study (SES) provided information on the frequency 
o^use of five methods of determining school eligibility for Title I thafcouldbe projected to 
national estimates. These data are presented in Table 3-1, along with the- unweighted NIE 
estimates for purposes of comparison. 



Table 3-1 



District Methods for Determining Eligibility of Schools for Title I »- 



National Projections From 219 SES Districts 



Method . 



Number 



: Percentage 



NIE Percentages 
(unweighted) 

-fx 



~ \~. =z 

] -%> 

Census data on family income 

i 


-> 

5,934 


44 4 




67 


AFpC enrollment ^ 


8.088 


, 605 


V 


51 - ' 


Fre^ or reduced-price 
breakfast counts 


1714 v 


If 

128 


t. 

4 

\ 


66 




Free br reduced-price 
luncmcounts 


'10,450 


78.2 








Number of neglected Or 
delinquent children , 

0 \ 

Other economic data 


4,084 


30.6 


f 


34 



The discrepancies between the weighted SES data. and the unweighted NIE data (based on 
responses obtained in the same $choot year, but from differently defined samples: the SES 
sample jncludes all. districts with grades 1*6? the NJE sample* includes only districts par- 
ticipating in Title I) are considerable. Faulty assumptions about how^istricts determine 
school eligibility could result in inappropriate use <5f the NIE estipiatesivDur projected 
percentages, .adjusted to consider only Title I districts, as £JIE jjjfd, indicate much less 
reliance on census data (44.4 percent vs. 67 percent for NEE) and much mor^ reliance drt 
counts of free or reduced-price meals (the projected estimate is 78.2 percent for breakfast 
and/or lunch counts, compared to NIE's '66 percent). * 



Very few of thje §ES districts used drily pne method fo\^ determining school eligibility (the 
reason the percentages in Table 3-1 do not add: to 1Q0)/ so we also projected the incidence 
of comb .atiorjs of hriethods. Table 3-2 presents the projected frequencies and percentages 
of all single methods and combinations of methods, based*on the SES districts. (Because 
each respondent is counted jn only' one method combination, the percentages in Table 3-2 
do add to the projected percentage of 93.5, the percentage of districts witj? grades in the 
1-6 range that participate in Title I.), 
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Table 3-2 



Methods and Combinations of Methods for Determining 
School Eligibility for Title I 



A 

' fr 
Method: C 
D 
E 


- Census data on family income 
■ AFDC^enrollment 

« Free or reduced-price breakfast counts 
« Free or reduced-price lunch counts 

- Number of N or 0 children serve* 


Projected 
Numbers 
of Districts ' 


Pmierffd 
Percentages 
of districts 




'D only 


2,369 


16.6 




B and D 


1,942 


13.6^ ■ 




A, B, D, and E 


1,490 


10 .4 




A, 8, and D 


' L362 


9.5 


* 


B only 


1,194 


8.4 




D and E 


656 


4.6 




A only " * 


594 


* 4./ 




A and D 


498 


.3'5 '" 




A and B 

# 


. 449 


3.1 




A, £, D, and E 


389 


2.7 




A, B, and E , , . - 


* 373> 


* 2,6* 




B, D, and E ^ 


> t ' 369 


2.6 




A* B, C, D, and E • \ 

>> 


335 






' C and D 


310 


2.2 : ' 




B, C, and D 


281 


2.0 




A, D, and E * * 


'238 


1.7. w 

• o 




A, B, C, and D 


. 159 ■* 


1 

1.1 




E only 


113 


' 0.8 




B, C, and E 


113 ' 


0.8 




C only 


59 


0.4 




A,C, and D. 


47 


/ * 
0.3 




B and C 


19 * 


0.1 




B and E * 


4 


0.0 




C, D, and E 


* 3 


90 



T 



Note Combinations that were not marked do not appear in this table, as their projected frequencies and 
percentages would be 0 We also project 926 *d i strict i or 6.5 percent of the nation s districts with grades 
In the 1-6. range, do not participate in Title I The percentages in the right column, therefore, add to 93.5 
percent of the nation's total 
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CHAPTER 4. THE DISTRIBUTION OF TITLE I SERVICES 



Ming-mei Wang, Ralph Hoepfner, Moraye B. Bear, Cerrie Smith 
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The results of a number of studies designed to inform Congress during the 

reauthorization hearings -for Title I are reported. Title I was found. to go to the 

poorest schools, with 'targeting' to schools differentiated* by region and 

urbamsm. Title I was also found to go to schools with higtier percentages of low 

..achievers. In the poorer schools, Title I services go more often to minority 

" " .» st " dents ' a * would be expected from the average poverty and achievement status 
of minorities. 

• Before the reauthorization of ESEA Title i in 1978, the U.S. Department of Education 
requested mformat.on on the current status of the program. An evaluation of the current 
status would lead to recommendations for changes or additions in the law that would alter 
the program in the future. Much of the attention of the hearings on the reauthorization was 
expected to focus on the politically and educationally sens.tive ,ssue of furids allocation 
Because considerations such as poverty, region, urbanism, minority status, and school size 
all play a role" in decisions about allocations, the Department of Education wanted 
comprehensive national information about them. The various data bases of the SES were 
used for information needed in a manner more focused on poficy needs than had been 
previously presented in the .technical reports from the study. We therefore focused- our 
analyses.on the issues of poverty, achievement, minority status and school size. 

SCHOOL POVERTY CONCENTRATION AND TITLE I* 

" J? 6 .™ 6 1 allocation formula prior to the 1978 reauthorization specified that a school's 
eligibility was to-be based solely on its proportion of disadvantaged children. Districts rank 
their schools according to the proportions of children from families in poverty and then 

- offer-Title I programs in schools with the greatest concentrations of poverty. (Jt is only at the 
level of student selection that low achievement was to enter the "selection process as a 

• cons.derat.onv) Because the rankings were within districts, it was not necessarily the case 
that nationwide'the poorest schools-would participate in the program? 

" * * ( 

There are many reasons why this was so. For example^one district will have all schools at 
relatively low poverty concentrations; anPther district, with the same average poverty 
concentration might have, some schools that are very poor and others that are not poor at 
all. This second district, in its ranking and selection procedures, might not be able to offer 
services to-af? of the very poor schools. We would expect, .then, that some participating 
schools in the first district will have less poverty than some hot participating in the second . 
Even if the distribution of poverty across schools were the same in two districts, one district 
might offer services in fewer schools with more .intense programs while .the other served 
more schools with less intense programs. Choices of the grades to be served or the subjects 
to be taught also reduce the exact correspondence between' schools' participation in Title I 
and their poverty concentrations. 
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In the tables below, based on national projections from 4,750 respondents to the Principal 
Survey Questionnaire in 1975-76 (see v Hoepfner> Zagorski, and Wellisch, 1977, for more 
information), we look at Title I and school poverty. From Table 4-1 we can see that the 
correspondence between^ concentration of poverty and being a Title I school is reasonably 
large. At the low level of poverty, half the schools offer Title I services but at the high level 
almost 85 percent do. A « 

The supenmposition of region onto the poverty distribution of Title I schools can serve to 
identify those regions in which the targeting' of schools is more or less closely aligned with 
the intentions of the Title I allocation formulas/ Regions in which such targeting conforms 
well ought to have small percentages of low-poverty (0-19?/o) schools participating and 
much higher percentages of high-poverty (30-100%) schools. From Table 4-2 we can see 

Table 4-1 

Percentages of the Nation's Elementary Schools Offering Title I, 
^ by Poverty Concentration 



Poverty Concentration 


Title*! Schools 


Non-Title 1 Schools 




Total • 


0-19% poverty enrollment 


50 8 


49 2 




100.0 


20-29% poverty enrolment 


76 7 


' 23.3^ 




1000 


30- 1 00% poverty enrollment 


84 5 " 


15 5 




Voo.o 


Total 


67 5 


32.5 , 


\ 

V 


100.0 



Note Projected numbers supporting the percentages can be round in Table A of the Appendix. 



Table 4-2 



Projected Percentage of Schools Offering Title I, 
by Poverty Concentration and Geographic Region 



Geographic Region 


> Poverty Concentration 
0-19% 20-29%' 


oflkhools 
30-100% 


Regional 
/ Totals 


* 

New Eneland 


5J 9 


83.8* 


78 9 


66.4 


Metropolitan Northeast 


61 3 


868 


83.8 


' , 713 


Mid-Atlantic 


55 9 


71.2 


. 89 4 


71.0 


Southeast 


36 7 


69 1 


89.4 


78.1 


North Midwest 


61 1 


870 


78.9 


68.9 


South Central 


39 1 


* 73 6 


88.6 


73.3 


Cenfral Midwest 


63 5 


93 4 


81 6 


73 8 


North Central 


• 42 2 


76 8 


75 2 


55 4 


Pacific Southwest 


• 14.4 


50 1 


69 3 ' 


~ 38.7 


Pacific Northwest 


49 5 


73 7 


95.8 


%50 


Poverty Concentration Totals 


50 8 


76 7 


84 4 


67.5 



Note- Projected/iumbers of schools can be found in Table A of the Appendix. 
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that no region displays any reversal in the order of percentages over the poverty categories. 
Half the regions have relatively more 'mid-level' poverty spools offering Title I than 'high' 
poverty schools. The regions appearing to conform mostlo allocation intentions are the 
Pacific Southwest, Southeast, and South Central; while those appearing to conform least 
are the North Midwest, Central Midwest, and Metropolitan' Noftheast. 

Table 4-3 shows the correspondence between the proportion of poverty and the presence 
of Title I programs as it varies across seven categories of urbanism. Here one sees that the 
pattern across all degrees of urbanism is the same as the overall national picture, except for 
rural districts. In those districts there is a slight reversal in which a greater percentage of 
. mid-level poverty schools offer Title I than high-poverty schools. 

The three categories of cities, with populations over 50,-060 conform best to the alloca- 
tion intentions. * * - ; 

. . .. Table4-3 - g , ., 

Projected Percerftage of School* Offering Title I, 
by Poverty Concentration j«tf Urbanism ' 



. . V 

•Urbanism " 


• — ^— ^ : — 

Poverty Concentration of Schoo^ * 
0-19% 20-29% 30-100% 


Urbanism 
Totals 


^ty^of over 500,000 
City of 200,000-500,000 
City of 5O,00O-200,OQ0 
Suburb of city 
City under 50,000 
Rural area near city , 
Rura^area not Rear city 


14,3. • 
12.9 • 
* % / 18.0 1 
45.9 
57 2 
68.4 
66.2 


" 25.5 
48.1 
53.8 
66.6 
. 79.6 
84.6 
91.2 


74.2 
73.0 
78.8 
73.7 
91.2 
89.2 
"88.1 


56*.2 " 
49.2 
48.5 
51.5 >■ 

76.8 
81.3 


Poverty Concentration jBtals 


50.8 


76.7 4 


"84.4 


67.5 



Note' Discrepancies within and between tables are due to rounding errors. 



STUDENT ACHIEVEMENT, SCHOOL POVERTY, AND TITLE I 

' \ 

Achievement and CE. After schools are identified as eligible to offer Title I programs, 
they are to provide services (purchased with Title I funds) to the educationally most needy 
students Due to*ie sequential nature of the process for getting funds and services targeted 
to students in various sites, and due to the fact that the criterion changes from an economic 
to an educational orienta'tion, not all of the lowest-achieving students in the nation will 
participate For example, such students may attend schools that do not offer Title I; they 
may not be the lowest of the low-achieving in their school; or they may be in a grade that is 
not participating at their school. 
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Although we may not expecf^n exact correspondence between a student's achievem&it. 
and participation in Title I, if students are grouped by their participation status, we would 
expect Title I students to rank very low. This expectation is borne out in Table 4-4, that 
presents unweighted data from the SES representative sample from the fall ohhe first year 
(1976) of the study. 

For both readin&and math, Title I students always have the^owest mean achievement, with 
non-compensatory students (in. schools offering compensatory services only from sources 
other than Title I) always having the highest. It may be noted that J he compensatory- 
education (CE) students always have the three lowest achievement means, and the non-CE 
students always have- the highest. Table 4-4 ajso presents some other information of 
interest • Jooking down the columns, we can see^quite clearty the resiitts of the difficulties of 
selectipg low achievers at grade 1 (wftere achievement tests are unreliable and achieve- 
ment is only partly academic). The standard deviations in Table A of the Appendix confirm 
this point The grade 1 difficulties seem to be not.SQ great for selecting participants in math 
CE programs, however. While there is a slight trend for the means of the CE students to be 
lower at the higher grades (cross-sectionally), the means of the non-participants seem to' 
remam fairly constant. It is this cross-sectionak phenomenon that has been labelled 'the 
widening achievement gap.' " S 

Teachers' Judgments of Need for CE. Table 4-5 presents information on CE participa- 
tion according^to teachers' judgments of students' needs tor CE, an acceptable student- 
selection criterion according to Title I regulations.. Our data are "based on judgments made 
for each student, separately for reading and math, by the homeroom teacher. We can see 
from Table 4-5 that nrOre Title I students are judged needy by their teachers than any other 
group of students, both for reading and math. The ranking of the groups by judged need is 
exactly the ^ameju the rankings by achievement in Table 4-4. 

We have converging evidence that, in general, the lowest-achieving students are 
participating in Title I. The evidence is that teachers' judgments are rather credible, 
although we cannot assume that they would remain so if achievement test scores were 
not also commonly used We s can also see from Table 4-5 that there are few trends over 
grade, except that more other-CE students in other-CE schools are judged to need CE as 
grade' increases. . ' 

•In a more theoretical v£in, we investigated the variables that explain (in a regression sense) 
whether a student is selected for Title I and whether a student is judged as needing CE by 
the classroom teacher. The practical implications'of this work concerned the advisability of 
recommending teacher judgment in lieu pf other (more obTective) methods for selecting 
participants. If, for example, we found that race/ethnicity was important in teachers' 
judgments of need, but not important in explaining actual participation (it wasn't found to 
be a useful predictor of either), regulations for selecting students to receive Title I could be 
altered to reduce bias that might have been indicated by the finding. Table 4-6 presents the 
average variances accounted for by each of -the stable predictors for both Title I participa- 
tion and judged need for CE. " . 

Generally the overall predictability of Title I participation is slightly lower-than the predict- 
ability of CE in general (See Wang, Hoepfner, Zagqrski, Hemenway, Brown, and .Bear, 



1978). When teachers' judgments are substituted as the crjtericpn to be predicted, and par- 
ticipation status'is not allowed to enter as a predictor, predictability is considerably higher. 

For reading judgments, it seems that the best predictors are current achievement, previous 
year's receipt of CE, and economic status; for math judgments, the best predictors are the 
same and in the same order. The relative differences in the percentages of variance 
accounted for in the prediction of the two criteria do not support art interpretation that 
selection and judgment j)y teachers are based on different considerations. 

Table 4-4 



Average Achievement of Students, by CE Status and Grade 



Reading 






Reading Percentiles 






v.c Matus 


Gr. 1 


Gr. 2 


Gr. 3 


Gr. 4 


Gr. 5 


Gr. 6 


Students in Title 1 Schools 














Title 1 students 


39 


26 


24 


25 


22 


22 


Other-CE students 


42 


• 41 


37 


36 


r 35 


33 


Non-CE students 


49 


52 - 


53 


, 52 


52 


53 


Students in Schools With Only Other CE 














CE students * > * 


42* 


33' 




33 


27 


26 


Non-CE^students 


54 


64t>* 


X 


61 


60 


59 


• 

Students in Schools With No CE 














All students 


48 


48 


* 52 




53 


51 


- * Math 






Math Percentiles 






CE Status 


Gr. 1 


Gr. 2 


Gr. 3 


Gr. 4 


Gr. 5. 


Gr. 6- 


Students in Title 1 Schools 














Titfe 1 students 


34 


29 


24 


29 


26 


27 


Other-CE. students 


45 


42 


36 


39 


40 


38 


Non-CE students 


49 


49 


48 


4 50 


49 


50 


Students in Schools With Only Other CE 














CE students * 


41 


40 


38 


36 


?9 


29 


Non-CE students , 


55 


59 


58 


56 


57 


56 


Students in Schools With No CE 














All students 


48 


■ 48 


50 


50 


52 


52 



Note Standard deviations, sample sizes, and means calculated to two decimal places can be found in Table B 
of the Appendix. — • 
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Table 4-5 



Percentages of Students judged by Teachers 
as Needing CE, by Their CE Status 



In Title I Schools 



Jrade 



Title! 
Students 



Other-CE 
Students 



Non-C€ 
Students 



In Schools with Only 

Other CE 
> * 



Other-CE 
Students 



Non-CE 
Students 



In Schools 
With No CE 

, All 
Students 





Needy 


Not 
Needy 


Needy 


Not 
Needy 


Nfedy 


Not 


Needy 


Not 
nieeoy 


Needy 


Not 
Needy 


Needy 

* 


Not 
Neeay 


1 

1 


OA 
OU 




40 


34 


lb 


Read ins 

74 


ol 


39 


16 


— =N — 

) 

y 

84 


23 


t 

77 




oo 
OO 




CO 


1 1 
41 


1:4 


-J6 


73 


27 


12 


88 


30 


70 


3 


87 


13 


52 


48 


21 


79 


70 






OO 


zo 


/ 1 


4 


86 


14 


51 


49 


22 « 


78 


72 


28 


15 


85 


25 


• 75 


5 


87 


13 


60 


40 


22 


78 " 


78 


22 


13 


87 


18 


82 


t> 


87 


13 


58 


42 


17 


83 


77 


23 


16 


84' 


21 


• 79 ' 


Across 
Grades 


86* 


14 


54 


46 


22 


78 


72 


28, 


14* 


V 
86 


24 


76 














Math 














1 


76 


24 


41 


59 


22 


78 


52 


48 


14 


86 


17 


83 \ 


2 


79 




■ 51 


49 


21 


79 


51 


49 


10 


90 < 


21 


79 


3* 


82 


18 


43 


57 


21 


79 


53 


47 


13 


87" 


25 


75- 


4 


75* 


25 


47 


53 


22 


78 


64 


36 


16 


84 


19 


81 


■5 


84 


16 


55 


45 


23 


77 


71 


29 


13 


87 


18 


82 K 


6 


80 


20 


44 


56 


20 


80 


68 


" 32 


13 


87 


18 


82 * 


Across 
t Grades 


79 


21 


t 

46 


54 


22 


78^7^1 


39 


13 


87 


20 


80 



Note: Sample sizes sXporting the unweighted percentages can be found in Table C of the Appendix. 




Achievement, Poverf^SMcl CE. ^ Funds are distributed to schools based, on poverty 
because it is believed tfiaKhdse schools will also have large numbers of low achievers! 
Therefore, it is important to note the correspondence between achievement and poverty at 
the school level. Students scoring at or below the 35th percentile on normed reading and* 
math achievement tests were*defined as 'low achievers' and, were aggregated to provide 
school percentages of low achievers. 4 Mean percentages were then projected for Title I and 
non-Title I schools by the three poverty categories. The projected means are presented in 
Table 4-7. The percentage of low achievers varies directly with concentrations of poverty. 
Title I schools have higher percentages of low-achieving students, except at the middle- 
category of poverty. This middle category is, quite likely, in the\g6neral range where 
many districts have to stop selecting schools for Title I, so the reversal of percentages is not 
tdtally unexpected. 
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Table 4-6 

Percentages of Variance of CE Selection and Judgment of 
Need Accounted for, by 'Student Characteristics 



Predictor 


CE Selection 


Teacher's Judgment of CE Need 


Reading 


Math 


- Reading 


Math 


Teacher's judgment of 










need for CE . 


26.3% 


17.3% 






Previous year's CE Receipt 


,4.4% . 


6.2% * 


24*9% ■ 


16.4% 


Pretest achievement " 










(percentile" s 5core) 


0.3% 


0.3% 


15.5% 


12.3% 


Other language spoken 










in the home 


0.1% 


0.1% 


0.4% 


( 1.5% 


Economic status index 


0.1% 


0.2% 


1.4% 


2.3% ( 


Total variance accounted 


31.1% ' 


, 24.1% 


42.2% 


32.5% 



Note Discrepancies are due to rounding errors. More complete statistical data on the regression analyses can 
be found in Tables D and E of the Appendix. 



Table 4-7 ; 

Percentages of Low-Achieving Students in Title I and Non-title I Schools, 
by School Poverty Concentration 



Title 1 
Participation 




Poverty Concentration of Schools 
0-19% 20-29% 30-100% , 


Across 
Poverty Levels 


Title 1 schools 




16.2 


23.7 


40.1 


28.6 


Non-Title 1 schools 




15.6 


25.5 


35.8 


20.3 


Across Title 1 status 




15.9 


24.1 


39.4 


25.9 



In Tables 4-8 and 4-9, we introduce region and urbanism into the^abalations of percent- 
ages^ low achievers in schools witrVdrfitetent degrees of poverty concentrations/it appears 
that the large cities and the rural areas in the New England, Mid-Attontic, South Central, 
Central Midwest, and Pacific Northwest regions contribute ^o the higher low-achievement 
percentages for non-Title I schools in the>middle poverty category, Comparison of figures 
for Title I and non-title I schools in the right-hand columns' of. both tables shows that no 
region or type of area has Title I schools with a lower mean concentration of low achievers 
than non-Title I schools. To the extent that this comparison indicates how well targeting of 
Title I to schools with low-achieving students has been, the Pacific; Southwest region and 
the very large citiesfedo the bestjob, while the North Midwes^and Central Midwest regions 
and iuburbs'do the worst. 
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Table 4-8 



f - 



1 



Percentages of Low-Achieving Students in Title I and Non-Title I Schools, 
by Geographic Region and Poverty Concentration 



Region 




Poverty Concentration of Schools 


Overall 


Title 1 Participation 




0-19% 


20-29% 


30-100% 


New England 
* Title 1 schools 
' Non-Title 1 schools 




* 16.T 
^ 13.3 


- , 18.6 
19.9 


29.7 


21.9 
16.3 


Metropolitan Northeast 
Title 1 schools 
Non-Title 1 Schools 




13.5 
11 2 


26.9 * 
12.3 


'40.7 * 
29.9 " 


25.1 
J4.3 


Mid-Atlajitic 
TitleTschools 
Non-Title 1 schools 




15.0 S 
16 4 


24 } 
32 1 


39.4 
- 33.8 


28.1 
21.2 


Southeast 
Title 1 schcfcls 
Non-Title 1 schools 




29 2 » 
19.3 


31 9 

29 2 


• 

43,0 
42.9 


40.6 
29 3 


North Midwest 
Title 1 schools 
- * Non-Title 1 schools 


V 


K 14.6 
150 


22.0 ^ 
22.4 


36.4 ' 

35.5 \ 


2K3 
.18*4 


Sou|h Central 

TitleM schools 

Non^Title 1 schools . 

• 




17 2 
16 6 


23 8 
28 8 


$9*5 

" , , 36.7 


23.8 


Central Midwest 
' Title 1 'schools 
Non-Title [ schools 




• 

U6.4 
15.5 


, - 23 2 


32.2 

29.5 * 


21.6 
18.1 


1 North Central , . 

Title 1 schools * 
* Non-Title schools 




' - J8.2 - t 
12.4 . 


184 ( 
. ;16.T 


* 35.3 * 1 . 
. 22.8 ; 


22.8 
13.9 


Pacific Southwest ,c 
- Title 1 schools 
Non-Titte I schools * 




< 

23 6: 
1 175 


32.1 
'27 2 


51 4 
41 7 


41.8' 
22.8 


Pacific Northwest 
Title 1 schools 
Non-Title schools 


e» 
\ 


16.8* 
19^. 


17.8 
49.9 


39.4 
• 20.0 


i 

24.8 
- 19.5 

% 


National Totals v 
Title 1 schools 
Non-Title 1 schools 




' 162 
'15.6 


■* * 

* *23.7 
55.5 


40.1 

'35.8, 


28.6 
20.3 



"Note Projected numbers~"ot schools can be found in Table A of the Appendix^ 
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Table 4-9 



Percentages of Low-Achieving Students in Title I and Non-Title I Schools, 
by Urbanism and Poverty Concentration 





> 

Urbanism 


jr Poverty Concentration of Schools 






Title 1 Participation * * 


/ c °- 19% 




30^100% 


wveraii 




20-29% 


—X r 


City of aver 500,000 




• 




< 




Title 1 schools 


\ 17.3 


29 6 . 


57.1 


53.7 




Non-Title 1 schools 


17.0 

• 


30.9 


39 3 


zo. 5 


City of 200,00a 500,000 












Title 1 schools 


24 3 


26 9 


. 48.9 


r 43.4 




Non-Title 1 schools 


198 


23.8 


35.3 


24.7 


City of 50,000-200,000 












Title 1 schools 


19.6 

• 


30.6 


40 5 


35.6 




Non-Title 1 schools 


. 15.5 


24.6 


„34.1 




Suburb of city 












Title 1 schools 


14.0 


24.8 


33.4 


18.9 . 




Non-Title 1 schools 


• 139 


k 24.7 . • 


45.1 


16.8 


City under 50,000 


- 










Title 1 schools 


. 14.8 


21.8 


3^0 


24.7 




Non-Title 1 schools 


15.6 


22 8 ' ' 


34.4 


1(5.7 


Rural area near*city~ 












Title 1 schools 


176 


21 7 


36.§ 


i 24.1 




Non-Title 1 schools 


1,6 i 


30.1 


35.5^ 


\ 20.2 


Rural 'area not near city 




m 


i 






Title 1 schools 


18.1 


24 3 


38.4 


29 7 




Norr-Title I schools 


16.4- „ 

* <r 


26.0 


31.7 


21.8 


National Totals 












Title 1 schools 


16 2 


23.7 


40.1 


28.6 




Non-Title 1 schools^ 


15.6 


25.5 ' M 


35.8 ' 


20.3 . 



Note Projected numbers of schools can be found in Table A of the Appendix. 
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Tables 4-8 and 4-9 tell us something about how well Title I is targeted to schools in terms of 
the low achievement of their students (a criterion that does not directly*enter into school 
selection), but tells us nothing about the students who actually participate in Title I. To get a 
picture of how well CE seaices are targeted to students, we altered our concern to the 
percentage of regular achievers (scoring above the 35th percentile) who participate in CE. 
We limited our search to the 213 CE schools in the first year of the study. The percentages 
of schools irreach region or area of urbanism with more thart 10 percent *of their regular 
achievers in CE programs are presented in Tables 4-10 and The data in these tables are 
unweighted, and based on small samples, but they do reveaf some trends. 

% V 

Tn almost all cases, there is a large percentage of Title I school^vn the 'more-than-10- 
percent-of-regular-achievers-in-CE' category. Although we canno^/ale out the possibility 
that Title,! services are just badly mistargeted, we believe thaU*Ke bad record for Title I is 
largely caused by the nature of the CE categories and the programs with which it is being 
compared. Our category of Title I schools includes schools that also have otherCL, so the 
poor .targeting may be worsened by those additional CE programs, Other-CJtf programs, 
when they are not layered on top of Title I (as they aren't, by definition of our catergortes) 
tend to be much smaller than Title I programs, and therefore serve fewer students. WeV 
woultl expect that smaller programs are easier to target well, even if their selection criteria 
are not as clear as those of Title I. Our rationale is confirmed, for example, in Table 4-10, 
where the Pacific Southwest hai the worst targeting for non-Title I schools. This* region,, of 
course, contains California, which has one of the most extensivestate CE programs in the . 
nation Y Other things being equal, the larger the program in the School, the more we can 
expe^to see mistargeting of CE services to regular achievers. ^ 

Table 4-10' 

Percentages of Title I and Other-CE Schools, by Region ^WitfvJttore Than 10 
Percent of Their Regular Achievers Participating inCE Programs 



Geographic 




Reading * 




Math 


Region 


Title i 


Other-CE 


Title 1 


Other-CE 


New England , 


69 


25 


0 


0 


Metropolitan Northeast 


" 72 




. 50 


0 


Mid-Atlantic 


76 


18 


47- 


17 


Southeast 


35 


50 


30 


0 , 


North Midwest 


33 


29 


33 




South Central 


44 




✓ 

37 


' 0 . 


Central Midwest 


f 47 


20 


41 


, 20 


North Central / 


47 


20 


•13 * 


20' 


Pacific Southwest / 


64 


63 


73 


63 * 


Pacific Northwest / 


58 


0 


25 


0 



/ Note Sample sizdfc supporting the unweighted percentages can be found in Table F of the Appendix 
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Table 4-11* 

Percentages of Title I and Other-CE Schools, by danism, With More Than 10 
Percent of Their Regular .Achievers Participating in CE Programs 




Urbanism 



Reading / Math 



Title I 



Other-CE 


Title 1 


Other-CE 


0 ' 


,80 


0 * 


40 — 


50 


40 ST 


8 


21 


0 , 


14 


23 


14 


1 33 


i 24 


8 


29 


35 


29 


; 67* 


41 


67 



City of over 500,000 80 

Cijy of 200,000-500,000 * \. 50 

City of 30,000-200,000 • 50 

Suburb of City 4 _ 15 

City under 50,000 56 

Rural area near city . • 47 

Rural area not near city ,59 



Note Sample sizes supporting the unweighted percentages can be found in Table F of the Appendix 

Anoth*/ way to gain an understanding of the nature of schools with different success in 
targeti/g CE to low-achieving students is to calculate averages of certain School 
characteristics for each category of targeting sUccess and then to compare the averages to 
see if there are distinct differences among the categories. Eight school characteristics are so 
treated in Table 4-12. Keeping in mind that the leftmost cokjmn signifies 'good' targeting of 
CE (feyv regular-achieving students participate) and the rightmost column signifies bad 
targeting, we can see that: ■ 



schools in larger districts tend 'to serve more regular achievers; 

poorer schools and schools in poorer districts tend to serve more regular achievers; 

J) •> schools With higher rates of parent/commqnity involvement tend to serve more 
0 ' ' regular achievers: and I \ • . 

• schools with more minority students tend to serve moreVegular achievers. * 

Differences according to schools size and expenditures are not strong, except that schools 
in the highest category of poor targeting have very low per-pupil CE expenditures (probably 
caused by a greater n.umber of participants overall, who serve as the divisor in creating the 
per-pupiMndex). / 

MINORITIES, SCHOOL POVERTY, AND TITLE I 

So far in pur consideration of the distribution of Title I, we have addressed only one educa- 
tional concern-achievement. Issues of poverty, region, and urbanism are political. The 
minority status of participating students is another gftlitical concern. Even though Title I is 
not; pncna facie^ a minority program (and such considerations do not appear directly 
in the la^vs or regulations), to the extent that minorities reside in areas of high poverty,, and - f 
-to the extent that they are low achievers, we would expect their participation rates to be v 
higher than for non-minortty students. The students we classified as being minority include 
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^ Table 4-12 

School Characteristics for Title I Schools With Various Percentages 
of Regular-Achieving Students Participating in CE Reading Programs 



Percentage of Regular Achievers 
School Participating in CE Reading Programs 



Characteristics 


0-10% 


11-20% 


21-40% 


41- 


100% 


Number of elementary schools in district 


13.3 


12.3 


45,4 




49,3 


District's percentage of poverty students 


24 4 




32 1 




31.9 


School enrollment in grades 1-6 


306 


313 


294 




329 


Parent/community involvement (ipdex) 


'39 3 


41.2 


41.8 




48,1 


School's percentage of minority students 


19.4 


19,3 


40.2 




49,2 


School's percentage of poverty students " " " — 


10.9 


14,7 


23,7 




24.3 


Currenfcper-pupil expenditures- ♦ 


$1,170 


$1,274 


$1,179 


, $ 


,153 


Per-puftl CE "expenditures 


$ 353 


$ .359 


$ 362 


$ 


248 



'Note Standard deviations, sample sizes, means calculated to two decimal places, and descriptions of the 
expenditure data can be round in Table C or the Appendix, 

Amefican,lndian, Asian American; black, Hispanic, and all other non-white^. The pertent- 
age of minority students in each sekjool was calculated as the sum of the principal's' 
estimates of the percentage of students in each component racial/ethnic group. 

From>Tabl$ 4-13 we can see clear evidence from the rightmost column that minorities do 
Mend to attend schools in higher poverty areas. In the lowest poverty area, Title I schools 
tend to have lower concentrations of minority students than non-Title I schools, but that is 
reversed in the higher poverty areas where Title I schools have higher concentrations of 
minorities. (We could conjecture that minority students in low-poverty areas are children^ 
in upwardly mobile -families who excell their non-minority neighbors in motivation 4 and 
achievement.) When these percentages are further analyzed by region, as in Table 4-14, we 
carrsee *h£ almost universal trend that lowipoverty Title I schools havejower concentra-* 
Hons of mirto'fity students th£n their non-TitleJ counterparts," but the reverse is trpe at the 

\ . Table 4-13 • ' 

v Projected Mean Percentages of Minority Students in Title I and 
Non-Title I Schools, by Poverty Concentration 



Poverty Concentration 


Title t Schools 


Ncfn-Title 1 Schools 


Overall 


0-19% poverty enrollment 
20-29% poverty enrollment 
30-100% poverty enrollment 


5.3 ~ 

10 3 . 
* 40 5v % } 


8.8 t 
20.3 
'35.8 


i : 

7,0 * 
12,7 
39.8' 


% Overall Poverty Levels 


' 224 


1A9 


20.0 ■ 



Note Projected numbers supporting the percentages can be round in Table A of the Appendix. 
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Table 4-14 

- Percentages of Minority Students in Title I and Non-Title I Schools, by- 
. • Geographic Region and Poverty Concentration of Schools 



1 

Region m * 
Title 1 Participation 


* 

Ppverty Concentration of Schools 


Overall 


>. 0-19% 


20-29% 


30-100% 


Poverty* 


4 ^New England 
Title 1 schools 
Non-Title schools 


2.3 
3.0 


T 7 

56 • 


22 8 
19.1 


10.1 
6.5 


Metropolitan Northeast 
Title l.schools 
Non-Title 1 schools 


5,6 
5.4 


1 6.5 ' 
33 3 


55 7 
47 4 


14.0 


Mid-Atlantic 
Title 1 Schools 
Non-Title 1 schools 


40 
9 5 


» 

99 , 
27 7 — 


/ 36 5 

S 30 9* 


20 5 
153 


Southeast 
Tite 1 schools 
; N^Title 1 schools 


16.6 
14 3 


148 
17 7 


38.2 
t35 4 


33.7 
222 


North Midwest 
"Title 1 schools 
Non-Title 1 schools 


3 5 
5 9 


69 
15.8 


37 0 
37.3 


12.3 . 
11.0, 


South Central 
Title 1 schools 
Non-Title I schools 


12 9 
14 3 


* 19,6 
25 5 


50.6 
47.5 


40.2 
, 24.8 


Central Midwest 
* Title 1 schools 
" Non-Title 1 schools 


3.4 
3.5 


*5.9 . 
1.4 


26.3 

1 O j 


9.7 
5.4 


North Central * 
Title 1 schools 
Non-Title 1 schools 


5.3 
4.3 


9,4 
139 


32.6 
25.4 


13.7 
7.6 . 


Pacific Southwest 
Title 1 schools 
Non-Title 1 schools 


18 '4 

T6.7 


25.4 
26.9 


53.9 

41.1 " 


22 7 
• 22.1 


Pacrfrc Northwest 

v * 

Title 1 schools 
Non-Title 1 schools 


'5.3 
' -59 


76 
6.4 


28.0 
0.0 


12.8 
5.8 


National Totals * 
Title 1 schools 
. Non-Title 1 schools 


5.3 
T 8.8 


10.3 
20.3 


40:5 
• 33.8 _ 


22.4 
14.9 



Note Projected numbers of schools can be found in Table A of the Appendix 
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high-poverty level. The major exception occurs in the North Midwest region, where the 
minority concentrations for Title I schools are always lower than for non-Title I schools. If 
the minority percentage for non-Title I schools is subtracted from that for Title I schools 
(rightmost column of Table 4-14), although the size of the difference depends in part on the 
overall regional minority concentration, we can still see that schools with high concentra- 
tions of minorities are most served by Title I in the South Central and Pacific Southwest 
regions, relative to non-Title I, and least served in the North Midwest. 

Table 4-15 presents a\&&ge concentrations of minority enrollments over the seven 
categories of urbanism It is clearly in the cities, at all levels of poverty, that Title I schools 

Table 4-15 

Percentages of Minority Students in Title I and Non-Title I Schools, 
by Urbanism and Poverty Concentration of Schools 



Urbanism 


Poverty Concentration 


of Schools 


O^all 


Title 1 Participation 


' 0-19% 


20-29% 


30-100% 


Poverty 

* 


City of over 300,000 
Non-Title I schools 


33 2 
21 6 


43 9 
39 8 


57 4 


OA \ 

39 1 


City of 200.00a 500. 000 
Title 1 schools 
Non-Title 1 schools 


22 8 
14 2 


29 1 
18 4 


67 7 
44 2 


57 9 
'23 0 


City of 50,000-200.000 
Title 1 schools 
Non-Title 1 schools 


13 7 
10 2 


25 8 
22 7 


50 2 
406 


40 7 
170 


Suburb of city 
Title 1 schools - * 

Non'Title 1 schools 

j * 


5 6 
84 


18 » 
23 1 


39 6 
36 6 


13 1 
11.3 


City under 50 000 
Title 1 schools 
Non-Title 1 schools ' 


4 4 

78 


90 
14 9 


33 6 
25 9 


17.6 
107 


Rural area near city 
Title 1 schools * 
Non-Title 1 schools 


3 7 
3- 


9 5 
10 7 


28 1 / 
27 7 J . 


12 1 
7 5 


Rural area not near city 
. Title 1 schools 
Non-Title 1 schools 


4 4 

5 1 


4 9 
10 8 


26.0 
163 


15.3 
89 


« 

National Totals 
Title l^chool? 
Non-Titrbj schools 


5 3 
88 


10 3 
. 20 3 


40 5 
35 8 


22 4 
149 



Note Proiected numbers ot schools can be found in Table A ot the Appendix 
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have*high^fhinonty enrollments, 'absolutely yvtfh regard to overall populatWexpectations, 
and relatively with regard to non-Title l,scFiools. At the highest poverty level, regardless of 
urbanism, Title I schools enroll higher* percentages of minorities than non-Title I schools, 
but the same is nor true at the lowerpqverty Jevels. • 
I \\ < 

SCHOOL size ; * . \ 

'The4ast characteristic we look at with.respe'ft to schools that offer Title Lrs their size. On* 
loag-starYding requirement incorporated into Title I law and regulations is. termed 
'comparability ' It requires that sSPvises provided by state and local funds be comparable 
throughout the participating district (with certain exceptions;. Comparability makes it 
impossible for districts to juggle funds'* and services among schools so as, to effect a 
redudion in state or local support to schools that /eceive Title I funds resulting, for 
example, in a lowering of the property tax rate at the expense of the Title I grant. One of the 
problems with the id£a of comparability is that provision of the same services may involve 
different costs for small or large schools? or for schools. in pear or remote locations. 
Exceptional schools.' as we will calf' them, result in complications for districts, that, -for 
example have a few small arvd remote schools. To maintain exact comparability the district 
would have to make costs equ$ in all similar schools, either to*the h^her costs of the 
exceptional schools, thus raism^titstricf tax requirements, or to the' lower costs of the large 
and near schools, thuj making- it impossible for the small ancUemote schools to provide 
adequate educational services. • . , 

In the example above, we discussed s&ool size and remoteness as contributors' to district 
problems in meetj^comparability requirements. Because school size is easier to assess 
than remoteness, TO. Congress wanted information on the distribution of sm#ll (less than 
100 students) and large schools, so it would have an idea of the results of writing an excep-' 
tfon tnto the law whereby small schools tould be excluded from district" comparability 
standards In Table 4-T6 we tabulate ^percentage of each region's schools that are small, 
and in Table 4-17 we do the same for the categories of urbanism. We also report the 
percentages of students enrolled in the small schools. , " 

From Table 4-16 we can^see that the North Central region f\as the highest percentage of 
small schools, so the districts in that region could expect to see some simpfification of their 
comparability reports*. The Pacific Northwest, Central Midwest, and New England regions 
also have large percentages of small schools, *As expected; Table 4-17 shows rural areas as 
having higher percentages of swiall schoc^. 

When we look at the distribution of large schools (enrollments of 100 or morf) over region 
or urbanism and poverty, as in Tables 4-18 and 4-19, we can see that comparability 
requirements (expected to fall most heavily on districts with large and poor schools) will be 
greatest for the Southeast and South Central regions and for the* large cities where thensrls 
simply greater poverty, and *illbe Jeast for rhe North Central andVPacific NortSwest 
regions and for Subdrbs and/ural areas near cities. % % 
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Percentages of the Nation's Elementary Schools That Are Small 
(Enrollments in Grades 1-6 Less Than 100) and of 
Their Enrollments* by Geographic Region 



Region 


Percentage of Schools 


Percentage of Enrollment 


New England 


16 1 




33 




Metropolitan Northeast 


0.6 




0.1 




Mid-ArJantiC * * ' 


56 




1.2 


* 


Southeast 


35 




0.7 




North Midwest . 


5.0 




1,1 




• 

South Central 


11 7 




.— 23 




Central Midwest 


16 1 




4.2 




North Central 


36 3 




7 2 




Pacific Southwest 


72 




1 2 


A 


Pacific Northwest * * 


19 2 




3 6 * 




Total 


9,3 * 




1.7 





Note Projected numbers on vvhich the percentages are based can be tound in Table H ot the Appendix. 



Table 4-17 . 

t 

Percentages of the Nation's Ejementary Schools That Are Small 
(Enrollments in Grades 1-6 Less Than 100) and of 



* * 


Their Enrollments, by Urbanism 


«» 


Urbanism 


Percentage of Schools 


Percentage of Enrollment 


City ot over 500,000 


08 > 


0 1 


> C>ty ot 200,000-500*000 


v 25 


0 5 


City or 50,000-200,000 


1 5 


0.3 


Suburb of city 


16 " 


03 


^ity under 50 000 


37 


08 


Ruraf area, near city - 


10 3 


-2 3 


Rural area not near city 


26 5 


6 7 ' < 


Total . 


9 3 


*<7 



No(e Projected numbers un Ahich the percentages are based can be found an Table 1 orthe Appendix 
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Table 4-18 



Percentages of Each Region's Schools That Are Large 
(Enrollments of 100 or More in Grades 1-6), 
by Poverty 



Region 



Poverty (Free Lunch) Concentration 





0-25% 


T£ TITO/ 

xD-JJ To 


36-50% 


51-75% 


76-100% 


New England 
Metropolitan Northeast 
Mid-Atlantic 
Southeast 
-North Midwest 


58.9 
69.2 
54.7 
2>8 
' 73 3 


1 

O. 1 

7.7 
14.3 
18.1 

8.7 


7.7 

7.7 
H.6 
21 .0 

6.5 ♦ 


5.6 
- '7.9 
10.1 
J9.3 
3.1 


3.6 
6.8 
3.8 
12.5 
3.4 


South Central 
Central Mjdwest 
North Central 
.Pacific Southwest 
Pacific Northwest 


34.0 . 
63.2 
51 6 
58.8 
65.4 


130 
*9 1 " 
6.4 - 
* 10/2 
8.0 ' 


17.5 
5.9 
7.2 
8.1 
4.7 


14.2 
3.2 
1.7 

10.2 
3.8 


9.7 
2.5 

1.9 

5.4 
2.2 


Total 


) 55.1 


11 0 


. 10.4 


8.5 


5.7 



Note Pro,ected numbers on wh.ch the percentages are based can be fou°nd ,„ Table j of the 
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Table 4-19 

Percentages of Schools liTEach Category of Urbanism That , 
Are Large (Enrollments of 100 or More in Grades 1-6), 
by Poverty 



« • 



Urbanism 




Poverty (Free Lunch) Concentration 

* * 




1 : r 1 


0-25% 


26-35% 


36-50% 


51-75% 


76-100% 


City of over 500,000 
City 6^00,000-500,000 
City of 50,000-200,000 
5urburb of city 
City under 50,000 
Rural area near city 
Rural area not near city 


28.6 
45.5 
,55.0 
85 4 
60.7 
63.8 
36.3 


9.7 
11.4 
12.5 
6.0 
13.1 
10.4 
> 11.6 


13.0 

s 9.6 

* 14.1 

3.3 

12.0 

8.8 • 

1h2 
• 


18.2 
19.1 
11.3 
3J 
7.3 
5.4. 
8.8 


• 29.7 
11.8 
5.8 
06 
3.2 
1.2 
5.6 


Total 


55.1 


11.0 


10.4 


8.5 


5:7 



Note Projected numbers of which the percentages are based can be foOnd ,n Table K of the Appendix. 
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Table A 

Projected Numbers of Schools Participating in Title I, by Poverty Concentration, 
Geographic Region, and Urbanism v 



Poverty Concentration 



Geographic Reyjon 
Urbanism 



0-19% Poverty 



20- &% Poverty 



30-100% Poverty 



New England 
Metropolitan Northeast 
Mid-Atlantic 
Southeast 

North Midwest^ . 

Sobth Qeritral „ * 

Central Midwest 
North Central 
Pacific Southwest 
Pacific Northwest 



Title I 



Non- 
Title I 



Title*! 




, Non- 
Title I 



Title I 



Non- 
Title I 



128 . 
93' 

2* . 

381 

271 

316 
61 

148 - 
549 
171 ' 



1,144 
1,271 
2,238 
5,727 
2,290 

3,523 
813 
481 

1,392 
524 



306 
246 
265 

* 682 
611 

• 479 
184 
159 
617 

23 



Total 



Title I 



2,953 
'3,708 
4,693 
7,118 
9,432 

5,085 
3,287 
1,804 
2,386 
1,717 



Non- 
Title I 



1,492 
~ 1,494 
1,917 
1,990 
4,265 

1,857 

, 1J69 
1,452 
3,787 
925 



Total 



14,833 



14,360 



7,949 



2,417 



19,403 



3,571 



42,185 



20,348 



City of over 500,000 * 143 855 

> City, of 200,000-500,000 104 698 

City of 50,000-200,000 . 532 2,430 

Suburb of city, 3,264 3,849 

City under 50,000 " 4,787 3,586 

Rural area near city 2,503 1,154 

Rural area not near city , 3,501 1,789 



115 
184 
459 
711 
2,592 
1,078 
2,810 



336 
199 
395 
357 
663 
196 
271 



-2,298 
908 
2,235 
793 
5,341 
1,494 
6,333 



799 
336 
601 
283 
5-17 
182 
854 



2,556 
1,196 
3,226 
4,768 

12,720 
5,075 

12,644' 



1,990 
1,233 
3,426 
4,489 
4,766 
1,532 
2,914 



fetal* 



14,833 



14,360 



7,949 



2,417 



19,403 



3,571 



42 ,1 85 



20,348 
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Table B 



Mean Reading arid Math Percentile Scores, by CE, Participation and Grade* 



Student CE 
Participation 
Status 



Reading 
« 



Math 



Gr. 1 



Gr. 2 Gr, 3 Gr, 4 Gr. 5 Gr. 6 Gr. 1 Gr. 2 



Gr. 3 


Gr. 4 


^Sr,. 5 


Gr. 6 


24.37 


29.21 


25 92 


27.49 


20.84 


21.97 


20.55 


21.47 


1,006 


895 


819 


657 



Title ['student 
in Title I 
school 

Other-CE 
student in 
Title I school 



* Mean' 
SD - 
N A 


39.32 
24.97 
1,602 


26.02 
18.74 
1,762 


24.33 
17.86 
^ 1,880 


25.10 
* 18 14 
1,540 


21 97 
17.80 
1,423 


22.40 ' 

17.37 

1,163 


34.25 
23 53 
850 


29 35 
.23.05 
858 


Mean 

SD 

N 


JL99 
25 76 
815 


40.57 
27.63 
826 


36.84 
27.25 
^ 706 


35.67 
. 26.11 
682 


34.60 * 
25.46 " 
* 640 


32.76 
25.42 
724 


45.17 
*25.80 
73 V 


42^42 
28^61 
626 



Non-CE student 
m Other-CE 
school 

Non-CE student 
in Non-CE 
school 



/ 



35.62 
26*5 1 
532 



39.48 
27.16 
626 



39.78 
28.21 
539 



Mean 

S.D. 

N 

Mean 

SD. 

N 



• 54.41 
29 94* 

2786 

48.26 
30.37 
1,268 



64.18 
26.17 
2,441 

48 17 
28.13 



61.77 
26.87 
2,489 

51.75 
28.02 
1,160 



60.92 
27 5Q 
2,487 

52.50 
28.94 
1,184 



60 05 
,26.68 
2,640 

53.36 
29.62 
.1,250 



58.54 
27.50 
4,334 

r 

51 40 
28.80 
1,662 



54,65 
29.38 
2,957 

47.83 
29.85 
1,264 



58.91 
27.38 
2,716 

47.59 
28 32 
1,104 



58.36 
28.16 
2,657 



56.28, 
\29.29 



56.79 
28.35 
2)646 2,773 



49.64 50.22 5?.H 
28.00 28.02 29.71 
1,158 l\l83 1,247 



38.31 
27.83 
634 



Other-CE student 
m Other-CE 
school 


Mean 

S.D. 

N 


41 88 

30.17 
533 


32.88 
23 66 / 
588 


33.25 
24.58 
558 


32.55 
23.88 
527 


27.04 
20 61 

54:2 


26 22 
20 82 
541 


41.27 
28.67 
361 


39.97 
29.64 
306 


, 38.44 
27.75 
389 


35.65 

. 26.49 

362\ 
1 ' 


28.98 
24.71 
404 


^79 

22.38 
-i 

424 


Non-CE student 
in Title 1 
school 


Mean 

SD 

N 


49.24 
28.85 
6,613 


52.29 
28 44 
5,462 


52.71 
27.50 
5,278 


51 73' 

27181 

5,434 


51.84 
27 72 
5,622 


53 31 
27.87 
5,962 


48.73 
28.69 
7,411. 


48 61 
28.45 
6,545 


48.25 
28.01 . 
6,322 * 


49.72 
28.25 
6,124 


48.94 
27.92 
6,326* 


r 

49.50- 
'20.44 
6,546 



55.94 
28.80 
4,440 

*» 

52.2Q 

29.06* 

1,659 



*The reading and math scores were obtained at the beginning of the school year (Fall, 1976). 
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Table 



Numbers of Students Judged by Teachers bs Needing CE for^Reading and Math, 
by CE Participation Status and Crade 

' I ■ • ' ' 



Grade 



In Title J Schools 



In Schools with Only Other 



TitleJ* 
Stujientt 



Other- % CE 
Students i 



Non-CE 
Students 



ther-€E/ 
udenfs 



Other 
St 



Non-i 
Students 





Needy 



Not 

Needy! 



**>reedy 



Not 
Needy 



In Schools 
With No CE 

All > 
Students 



Not 
Needy 



Needy 



■ 1 






2 




\ 628 




\ 


1,670 
• 






1?344 


5 ^ 




1,297 


6 




1,030 




> 




Total 




8,332 



v. 



249 
22? 

187 f 
152 



393 , .464 
513 P ,353 
387 357 
361 346 
401*" 2fj7 
4261 i 313 



Reading 

5,089 
4,317 
4,324 
4,352 
4,501 
5,093 



337 ^15 
444 ^> 1^2 
397 

379' ' ISO 
427 120 
426 " 129 



448 ' 2,451 * 
300 2,203 
354 2,262 
398 ,2,166 
33& 2,359 

^ 707 _i699^*r 



306 
349 
336 
300 

345 



996 
802 
881 
912_ 
1,051 
1,337 



1,382 



2,481 2,100 f 

■ v ■ 



27,676 2,4]0— -94V 



2,543 



15,140 



1,872 5,979 



V 
■ 2 

3 

4 



692 * 21.5. --—3f5^ 45$ ^1,702 

704 ' v >8> „ * ,338 < 32fP 1,428* 

834 179 ' • 241 326 J, 394 

'681 229, * ^03. 347 1,381 

717 , 139 506 V ' 254 1,521* 

538 k 138 282 3^4 1,359 



Math 

• „ 

6,032 
5,388 
5,156 
4,917 ' 
4,992 
5,353 > 



86 

208> 
233, v 
290 
3 / 



V69 
150 
184 
129 
~12(5' 
138 



417 2,679 * 


'217, 


h085 


273 2,531 


' -243 


908 


350 ' 2,442 


299 


-918 


426* 2,305 


233 


979 


364 2,468 


231 




5^7^3,933- 


305 


1,377 


2,427 16,356 


J, 528 


6,323 




X166 



,087 



1,785 ■ 2,065 



8,785 31,838 



1,365 



890 



-82 



Table D 



* * 
Correlations, Standardized Regression Coefficients) and Percentages of Variances Reduced by Each Successive- 
Entry of Predictors of Student Selection for Title I Services for Reading and Math, by Grade 



Predictor 



C "* 11 ' C " d * 2 . Gr.de 3 " Cr.de 4 Cr.de 5 Cr.de 6 

Beta \V r- Bet. \V r- 4 Bet. %V f 



Bet. %V r- Bet. %V r- Bet. %vj - 



• * - ■ r^f — 

T u u Re.p*ing « * • 

Teacher judgment o* a \ * 

neodfcead.ngCE V 42 56 ■ 8 52 > 27 J . $6; ' 3> 5 52 Mb *7 3 52 ' Si 26 9 52 32 27 0 

^JPre*»oos year s t ' e ' < * 

~d,ngClW 22 .'JO .0 40 2r , 8 47 23 4 . . 46 " 24 4 2 50 ^ 60 51 ' 7 .' 

Reading achievement * * ' 

<0*,«mte«O,e, .. - 14,^04 "V- - 09 0 3 - 4.- 08 05^ -,, - 06 0 2 - 39 - 08 0 3 - 36 - 04 0 2 

Qher laoauage # ' j t * 

spoken mil* horn • 07 - 03 o3 04 - 06 0 J , / 09 - 0. 00 . 08 - 03 ' 00 00 0 0 08 ..0. OO 

Economic Mat us , * 

'"^ x • - 2 ' -° 7 °*~ . -0-' 00 - 24 - 0. 00 - 23 - 02 00 -22 02 00 -'24 - 02 00 



Multiple corfe*dtion 44 , 56 " ' 60° 



56 - - 58 39 



Sample v« . 8 ? 6Jb • , 7 101* ?0 84 6 847 ■ 6 937 7 32 7 ' ' 

Total \ vyance 1945 n 7^ " 3604 /, 75 33 27 . 34.271 



Math * / * f 



Teacher judgmenr ot , 

neediornuihCE # ,7 _ 28 1) 7 42 ^"32 I72> 45 , 29 20 J 40 25 .64 44 28 .9 5 4. 23 .6 7 

Previous year s * , 

.-haw 26 .5 2 3 ,4 2. 4 1 43 29 ~7 5 39 26 5 7 J* 3. 78 \ 46 35 .0 0 

Math achievement . / * 

lpe««*le«o«. , . ,«, 04 . -0 2 - 23' ^5 0 2 . 29 - 05 0 3 - 2*5 '- 07 0 5 - 29 - 03 0, \>3 -OS 0 2 

Other language * » . '« 

spoken ,n*e home 08 - 03 0 0 05 -05 0. .0 0 . 0 . 05 - 04 0 0 06 - 04 0 . 07 - 0. ' Ooi . 



Ecorxxryc status 

•odex - ?4 io 0 7 -18 -*02 0 0 " - 22 * - 04 0 1 



Multiple correlations 41 / 47 

Sample we 6 7$7 5 797 

Total % variance * T6-0« * 2 T 67 



20 - 05 0 2 * - 20 - 03 0 1 - 18 - 01 0 0 



53 48 ^ 53 .52 

5.731 5 403 % 5.368 5.862 < • 

28 34 22 80 ' 27 60 2*93 



O fl • / 

eric , i. *- , ' . . , 83 •• ; t 



* * Table E * 

Correlations, Standardized Regression Coefficienl^nd Percentages of Variances Reduced by Each Successive 
Entry of Predictors of Teacher's Judgment of CE Nfeed for Reading and Math, by Grade 



^ 6 C"de 1 ' Grade 2 Grade 3 Grade 4 Grade' Grade 6 

Predictor ~ * 



r Beta %V r* % Beta %V? V * Beta %V r* Beta %V r* Beta %V r' Beta %V» 



^ * - " * Reading. " ^ ^ 



Previous year > » ♦ % v v • 

reading (£ receipt JO 24 9 1 42 25 17 9 5 3 34 f 9 56 37 s 30 9 * ' 58 3> 33 3 \ 55 38 30 3 

Reading jc hie*-ement w 

(percentile score- 23 -In -4 9 - hi 50 24 9 - - 61 - 44 19 3 - 58 - 39 'l4 3 - 62 - 41 16 0 - 56 - 37 13 5 

Other language , "* ^ 

spoken in the home H 05 l J p * 03 0 2 17 05 0 3 * 15 - 02 0 0 19 .04 0 3 14 01 0 0 

Economic sjEalus * * ' ' 

inilen *8 29 "J -36 04 0 1 ^36 - 05 0 2 - 36 - 08, 0 4 - 39 - 07 0 3 - J5 - 07 0 4 



— • \i — 

Multiple correlation 
Sample size 
Total \ variance 


8 


166 


22 56 


'66 


161 " • • 
4 3 09 


69* 
% * 




4^3 


68 


6 847 


^sji 


71 


6 937 
\ • 


X — 

49 90 


67 


7,327 


44 30 


> " 


•> 












. Math 






















. .Previous year s * 


c* 

o 




« • 






























mj'h (. i tin eipt , 




2 3 


4 ] 


3* 


26 1 3 8 


41 


25 


16 5 


44 


33 

4 


19 2 


' 46 


30 


21 1 


• .43 


33 


18 8 


* «A 

Main achievement* 


if* 




'ft ' 






























'percentile m orei 




21 


■< 5 


43 


32 12 3 


50 


V 


14*9 


- 45 


- 32 


11 7 


- 55 


- 40 


17 1 


- 43 - 


- 29 








































Other Ungu.ige^ y 


is 


()5 
































spoken m the horn** 


2 0 


18 


tt*-^ i a , 


16 


05 


•2 J 


12 


- 01 


06 
i 


18 


05 


1 5 


13 


00 


08 


fc ononuTLatUN 


































indt x * , 




)*• 




- 34. 


- r> 18 


33 


- 11 


0 9 


- 32 


'» 


1 7 „ 


- 36 


- 12 


1 1 


- 33 


- 16 


1 9 


Multiple ( >r relations 
Sample si/e # - 
Total % variance 


50 


'57 


24 69 
■** 


34» 


5 '97 

29 68 


59 


5 731 


34 41 


58 

o 


5403 


33 16 , 


64 


5 368 


40 77 


57 
^> 


5.862 


\ 

32 03 4 



•r - umplc t'>rrt!atMjt"i oi the pretlutur with Till* ,1 partic i pat ion status ot 1 9*6- 7"* sthool year based only on students in schools participating in Title I tail non participating 
, students were considered as not receiving tb \ 

» ** I • 

**X\ - p*r<« ntjgt d\ .nkJitiunal varum t ^reduced by entry ut tht pffdiUur order ut ihe prediUors* determined fromiongin<il pTedictor orders with CE participation as the 
Q and baMtf c>n the entire SJS sample \ / ' J 



Table F 



Numbers of TitleJ and Other-CE Schools, by Region and Urbanism, With More 
Than 10 Percent of Tlieir Regular Achieyers Participating in CE Programs 



Region 
Crbanism 



Reading 



Math 



Title I 



Non-Title I 



\ew* England 
Metropolitan Northeast 
Mid-Atlantic 
Southeast 
\orth Midwest 
South Central 
Central Midwest 
North Central 
Vatirlc Southwest 
Pacific Northwest 



9 
13 
13 



Total 



84 



13 



Gtv or over 500 000 
Citv or 200 000-50}) 000 
Citv of 50OQO-2OO0OO 
Suburo ot otv 
X.X\ anjder 50 000 * 
Rural-area near ut\ 
Raraj area not near c it v 



F^tal 



/ 



2b 
8 
30 



84 



13 



Title I 



0 
9 
8 
6 

6 
6" 

2 
8 

3 ■ 



3 
3 
3 

II 
6 

21 



Non-Title I 



a 

0 

1 

0 




Virnf>)e-is based on 2 M *_hooU ,r the rirst vear or the studv that participated m an V CE program It can be 
seen Then mat <T ot fhe schooU had more than 10 percent -or "their 'regular-achieving students 
participating ,n CE reading programs and 64 had more than 10 percent participating in their CE math 
programs The remaining schools had fewer than IjQ percenter thejr regular-achieving students in the 
respective C\ program f - 



Table G 



Schoi^Wiaracteristics for Title I Schools With Vanous\ 
Percentages of Regular-Achieving Students Receiving CE Reading Services 



School 0 



Percentage of Regular Achievers , 
Receiving CE Reading Services 



v» llaidv.tCi DHL) 




0-10% 


11-20% 


21-40% 


,41-100% 


Total 


Number of 


Mean 


13.34 


12 30 


f ^5 40 


* 49.25 


• 

21.82 


elementary schools 


sp 


19 27 


, 22.56 


107.66 


* 

107.54 


58.55 


m district 


N 


73 * 


43 


25 


16 


1-57 


District's ' 


Mean 


24 40 


26.44 


32 '12 


31.94 


26.96 


percentage or 


S D 


20 90 


16.17 


•21 18 


14 76 


•19.28 


poverty students 


N 


73 


43 


25 


16 


157 


Schcfol 


JMean 


305 67 


312.56 


294 28 


328.69 


308.09 


enrollment 


. S 0 


' 161 54 


• * 236,67 


150 31 


235.01 


139.72 


m grades 1 -6 


N 


* 73 


43 , 


25 


16 


J 57 


Parent/ 


Mean 


„ 19 33 


o 41.16 


41 76 


• 48 12 


41.11 


community 


S D 


12 35 


* 16.31 


12 59 


18 91 


• 14 39 


involvement 


N / 


73 


4> 


i 25 


♦ 16 


157 


School's ^) 


Mean 




f 9 30 


40,16 


49.19 




percentage of 


S D 




31.57 


59.66 


39.48* 


33.71- 


minority students 


N 


73 


. "43 


25 


' 16 


157 


School's 


Mean 


10.92 


14 67 


23-72 


* !*24.25 


15.34 


percentage of 


S.D. 


10,35 


* 16 3*1 


23.35 


26.66 


17.38 


poverty students 


N 


' 73 


43 


. 25 


' 16 


•^57 

m 


Current 


Mean 


$1,170 


$1,274 


$1,179 


$1.1-53 


$1,198 


per-pupil 


S D 


$525 


. S476 ' 


$400 


$386. 


$479 


expenditures 


N 


.73 


43 


25 


16 


157 


Per- participant* 


Mean 


, $353 


$359 


' \ $362 


$248 


\ $345 


CE Expenditure 


5 D ^ 


$282 


$219 




$ 1 63 


$245 




N 


73 


$3 




16 


156 



Note Per part iciparff expenditures were obtained by dividing the reported total CE expenditures by the 
reported number of participants Valid expenditure data were not ayaijable for one Title I school. The 
index of parent/Community involvement has an'overall mean of 42 1 and a standard deviation of 14.6. 



Table H 



Projected Distribution of the Nation's Elementary Schools With Grades in the 1-6Hange 
and4(ieJ>Say-6 Enrollments, by Geographic Region and Sdiqof Size ; 



Region 



Small Schools 
(Less^than 100) 



Large Schools 
^(100 or, more) 



Total 
, * * 
Population 



Number 



Enrollment 



" Number 



Enrollment 



umber 



Enrollment 



9 New England • ^ 
Metropolitan Northeast 
Mid-Atlantic 
Southeast # 
North Midwest* 

^duth Central 
Central Midwest 
North Central 
Pacific Southwest 
Pari tic Northwest 



. 714 

32 
373 
315 . 
690 

812 
719 
1,180 
446 t 
508 



44 J 50 
3,009 
27,936 
23,213 
46,944 

52,791 
44,472 
47,184* | 
26,046 
25,488 



3,731 . 


1,096,448 


4,445 


1,140,594 


5,170 


^252,999 ' 


. 5,202 


2,256,0Q8 


6,236 


2;234,407 


6,610 


2,262,333 


8,792 


3,507,054 " 


*^ u '9,108 


3,530,259 


13,007 


4,251,762 • 


^ * 13,698 , 


4,298,699 



6J31 
3,738 
2,075 
3,726 
2,135 



2,206,679 
1,017,828 

606,228 
2,238,137, 

678,888 



6,943 
4,457 
* 3,255 
r 6,173 
,•2,643 



'2,259!462. 
1,062,297 
. 653,412 • 
2,264 J 78 
704,373 



Total 



• 5.790 



34,1,232 



56,741 



20,091,988 



1 



62,534 



20,431,590 



Note Discrepancies between totals and actual surns are due to rounding errors > * 
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Table I 



Projected Distribution of the Nation's Elementary Schools With Grades in the 1-6 Range 
and the Grade 1-6 Enrollments, by Urbanism and School Size 



Urbanism 



Small Schools 
(Less than 100) 



^Large Schools 
(100 or more) 



Number 



Enrollment 



Number 



Enrollment * 



' Total 
Popi 



otai 

ulatron - 



Number 



Enrollment 



Gty of over 500 000 


36 


3,306 


4.511 


2,308,943 


4,547 


2,312,248 


City of 200 000-500,000 


61 


4,761* 


2,368 * 


953,819 


2,42$ 


958,579 


City of 50.000-200,000 


97 


7,314 


6*555 


2,377,694 


6,651 


- < 2,385,007 


Suburb of city 


* 150 


12726 


9,105 ' J 
16,833 


3,681,238 




• 3,693,96 1 


City under 50,000 


• 652 


43,433* 


"5,692,448 . 


17,485 


5,735,872 


Rural area near city ^ 


679 


. 44,338 


5,927 


• 1,925,640 


6,607 


1,969,968 


Rural area not near city 


4,116 ' 


223^350 


11,442 


3,150,505 


15,558 


3,375,781 
















Total 








-w — — 


» 


— i — „ _ 


5.790 


341,228 * „ 


56,740 


20,090,287 


- 62,530 


20,431,400 



Note Discrepancies' between totals and actual sujps are due to rounding errors. 



EMC. 



8* % 



Table) 

Projected Numbers of Large Schools (Enrollments' of 100 or More in C 
and Their Grade 1-6 Enrollments, by Geographic Region and Po 



Les 1-6) 



< Repoh 




0-25% 


>• 


26-35% 


36-50% 


31-73% 


76-100% 




Total 


Schools 


Enrollment 


Schools Enrollment 


Schools 


Enrollment 


Schools 


Enrollment 


Schools 


Enrollment 

SI 


Schools 


Enrollment 


New tngljnd 


2 619 


"68 485 


J61 


105 00? 


344 


10665J 


248 


64,131 


160 


52 199 


3 "31 


1 0% 468 


MetrotJO^tan Northed 


3 6(U 


1 499 064 


399 


1 54 35" 




191 8"6 * 


41 ! 


215 935 


352 


191 81 3 


4 170 


2 253044 


Mid AtljntK 


3614 


i rs 353 




324 650 


"64 


& 264 558 


665 


1 261 415 


252 


107 511 


6 23" 


2 234 487$ 


Southed 
\ 


' J 348 


J 105 44} 


f 644 


65J 919 


1 909 


"02 564 


1 "54 


652 420 


1 13" 


392 °OS 


8 "93 


3 507 254 


North Midwest 


10O4! 


2 * 204 122 


1 190^ 


356 "42 


885 
* 


2"} 248 

r 


429 


164 922 


462 


252 ""9 


13 007 


' 4 251812 



South Cent Ml 


2 358 


890 219 


901 


302 009 


1213 


395 359 




^61 626 . 


673 


25" 5-3 


6 13! 


2 206 786 


Cent Mi Midwest « 


: 8i9 


'"99 18" 


405 


88 808 


262 


58 046 


$ 142 


42-235 


n 1 


29 5"3 


3 738 


1 017 846 


North Cenlwt 


1 681 


50" 168 


208 


55 416 
* 


t "1 


1" 640 


56 


12 %1 


58 


13 043 " 


2 075 


606 22*9 


PvKittc Southwest 
Pacific Northwest 


J631 
1 7J9 


1 41060! 
540 682 " 


630 
212 


24 3 197 
"4 029 


502 
125 


20) 680 
* 39 602 


632- 
10 


222 524 
5 981 


33*! 
59 . 


I8>" 


5 727 
2 135, 


2 238 ^81* 
678 889 



fotal - J4445 1><X)1 356 £893 2 358 121 
i 4 ^ 



6 478 2 251214 



5 3 32 2 004 1 37 



3 595 1 476 1 79 , 56 74 3 20 09 1 006 



Note fhe disuefwncies between to^K jmrf {he Jttu.il surr^re due to rounding errors 

^ - 

ERIC . 



'89 



• * 






• 


* 












V 


• 










• 




• * 




<* ♦ 


• 


Table K 








t 




• 


•i 


Projected Numbers of Urge Schools (Enrollments of 100 or More in Grades 1-6) 
and Their Grade 1-6 Enrollments, by Urbanism and Poverty * 




- 






Poverty (Free Lunch) Concentration 










Urtunivm r 

» 


0-25% 26-35% 


36-50% . 


SI-75% 


76-100% 




Total 




Schools Enrollment School* Enrollment 


Schools 


Enrollment 


Schools 


Enrollment 


. Schooh 


Enrollment 


Schools 


Enroflmeni 


\ * 
^C.tv 0* over 500 000 


1 102 601 921 441 216 '35 


589 


3!0 744 


<827 


438 5 56 


1 352 


t 741 075 


4 51f 


2 309 030 


Crfv or over 200 OtfVSOU 0(X) 


1 104 4-*_> 452 2" * T 14 92 5 


' 234 


100 285 


465 


203 095 ' 


* 287 


103 088 


2 368 


953,843 


C*v of 50 000- 200 000 


3 659 1 302 9'8 829 30? 631 


935 

* 


348 261 


T50 


285 759 


383 


139 144 


6 555 


2 377 770 


Wxjrb Of t'ty 


~<V* 3 211 '61 558 210 4 2 3 


? 305 


13-456 


285 


102 299 


53 


.19 328 


' 9 T05 9 


3 681 271 


tC»ty under St) (XX) 


10605 3 540 33' ' * 2 29b '94!') 


2 092 


'699 281 ^ 




478 014 


563 


180 782 


16 834 


5.692,588 


Rural area near c it> 

a 

RufdJ .wpd not near ctfv 


4 218 1406 505 68^ 204 071 

5 652 1 505 2)5 1 810 516 15' 


*#■ 
584 
-ft • 
1 7)9 


184 464 
4'0 7)) 


358 
l 365 


106 4 24 
390 001 


81 

87? 


24^6 

268 542 


5 928 
11 442* 


1 925,681 
3 150,666 


* ' TocjI 


)4 444 |>0M 294 6 89) 2 )58 105 


6 478 


2 251 210 


5 332 


2004 146 


3 595 


1 476 167 


56 742 


20,090.925 


Note The d»<< reparx ie% between totalv and aufat vjfm jre due^o rounding prrors 




• 










* 


* 






















✓ 


• 


















ERjC rs** 


* 


• 
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• • , -CHAPTERS. " 

THE EDUCATIONAL PROGRAMS OF CE AND NON-CE STUDENTS 

Ming-mei Wang 
Ralph Hoepfner 
Moraye B. Bear 
Cerne Smith 

> •* 

Title I students are offered the same number of days of instruction, but their 
absenteeism is slightly higher than that for regular students, resulting in the fact 
that they receive about four fewer days of instruction per year. While Title I 
students are receiving their compensatory educational 'services, -non- 
participating students recede other, but very similar services. When CE services 
are combated oh the basis of the number of years children participate, Title I is 
found to ha\e the greatest continuity. This may be because Title I is the most 
permanent of the CE programs, or because the'most needy children are-selected 
for it and they are the most lively to continue needing it. 

0 

The framers or the Title I laws and regulations have been very deliberate over the years not 
to specify what sea ices would be delivered in the name of Title I. General requirements 
exist to focus the services on academic areas and to ensure that they are supplemental, but 
Congress has been loath'. to infringe much more on local decision-making. While refraining 
from interference m the content or process of CE, Congress did want information on some 
or the more mechanical aspects of the services iha\ cbuld be useful in framing regulations 
on comparability Specifically, information was requested on length of schpol year, atten- 
dance rates non-CE activities wtfule CE participants receive their CE services' and continuity 

of CE services • , 

-> b 

Length of School Year One way that schools and districts can conform "to the supple- 
ment not supplant' requirement of Title I would be to extend the school year for Title I 
participants 'other ways might involve extending the length of the school day or furnishing 
summer-school instruction or improving the quality' 6f instruction). Average members of 
days in the^hool year are tabulated for the six CE participation groups in Table 5-1. The 
means are based on students in the first-year representative ^mple, and are based on 
students rather than schools in order to be consistent with attendance data to be discussed 
shortly, As can be seen, the meaAs all hover close to 177 days, so it is clear thaMitle ! 
students do not receive more #ays of schooling — 

Attendance Although the number of days that instruction is offered cal be administratively 
controlled to some degree, -the number of days each student receivesNnstruction is much 
fess controllable Table 5-2 provides average attendance rates that would.be helpful m de- 
termining whether Title I students receive the supplemental services they are ottered. *V£* 
can see that CE students, Title I students in particular, have slightly lower average attendance 
rates A difference of about 2 5 percentage points is equ^l to about 4 days of schoofmg. We 
can also see that attendance fs better in the higher grades. ' * 



t 



\on-CE -\ctnittes. The supplement not supplant' requirement for Title I presents 
schools v\ ith a dilemma if they attempt to adhere to it too strictly The .problem, of course, is 
what to do with the non-Title I students while the Title I students are receiving their sup- 
plemental Title I services Because schoofs have not yet adopted cryogenic techniques for 
controlling student activities, the non-Title I students invariably are doing something, and 
that something, if the school is being responsible to its charge, should b& educative— bat 
• not something that the Title I services would be supplanting. Information on what non-Title 
I students do not only gives us dataUn how the supplement' is being implemented, fl>can 
also serve to help us better to understand the results of evaluation studies. 

In Table 5-3 we can see that the most comm6n alternative activities for non-participants are 
in the same subject area, giving a strong impression that supplanting is actually taking 
place Less related activities, -such as study, self-selected activities, hbrary , work , / gym, and 
field trips are also common when considered as a class of activities not likely to give rise tp 
accusations that Title I services are supplanting other services. 

Continuity of CE Participation - Based on loud and public complaints by several large 
school districts that strict obedience to Title I rules for the selection of participants resulted 
* m a revolving-door program for many marginal achievers, Congress also wanted informa- 
tion on the continuity of Title I for individual students. This information would be useful in 
considering ^h^nges to regulations that would encourage continuity instead of rigidly 
enforcing 'nov^sy-quahfy, povv-you-don't' rules . . * 

o # 

. t • Table 5-1 



Mean Number of Days in the School Year, by CE Participation Status and Grade 



CE Participation 




Gr. 1 


Gr. 2 


* Gr. 3 


Gr. 4 


Gr. 5 « 


Gr. 6 


Tstle,. I student in 
Title I school 

^ther CE studer\tjn 
Title 1 school 


Reading 
Math 

f 

. Reading 
Math 


176 7 
1770 

176 8 
176 4, 


175 8 
174 6 

178 0 
177 1 


1.76 8* 
,1 76 6 

178 3 
177 2 


1778 
177 7 1 

177 9 
177 1 


178.2 

177 4 

178 0 
177 1 


177 5 
175 9 

177 1 • 
177 4 ' 


Other CE student m 
Other CE sci^ol 


Reading 
Math 


178 6 

179 0 


177 7 

178 3 


• ,1 78 5 


176 8 

177 2 


177 4 
177 2 


1770 
177,2 


Non CE student m 
Title 1 school 


Reading 
Math 


1 7b > 
176 \ 


177 0 J 
1770 


* 177 5 

177 6 


1774 
177 5 ' 


1 77 6 

178 1 


176 9 
177.1 


Son CE student m 
Other CE school 


Reading 
Math 


178 1 
178 1 


178 3 
178 2 


r8 * 

178 \ 
( 


^ 1 77 6 
1 77 5 


177 9 
177 9 


177 {f 
I77'5 


\on CE student in 
Non CE^sc hool 


Reading 
Moth 


177 0 
177 0 


177 3 
177 3 


177 2 
177 2 


175 4 
175 4 


175 6 
175.6' 


175,8 
175 8 



Note Standard deviations sample >*zes, totals and mean«> calculate^! to two decimal places can be found in 
table A ot the Appendix 



i 

Table 5-2 , 

Mean Attendance Rates, by CE Participation*Status and Grade 

* « 





CE Participation 




Cr. 1 


Cr. 2 


Cr. 3 


Cr. 4 


Cr. 5 


Gr. 6 




Tltip 1 cttlHpnt in 
i i jiymcill 111 

Title J school 


Dn -\ r\ i i-i n 

KcdOl ng 

Math 


91.6 


94.6 
93.5 


94 1 
93.8 


^,94.6 
94.3 


94.4 
94.2 


94.9 
94.1 


9 


v-/tncr -v^c MUQcnl in 

Title 1 school 


Da l/ii rs/i 

KGdUing 

Math 


93 o 

94 1 


94.6 
95.0 


94.7 
94.3 


95.2 
95 1 


94.1 
94.2 


95.0 
, 95.5 




t/thpf.i r cti irlont in 

Other-CE school . 


* Reading 
£1ath . 

4 


% 94.6 


95.8 

,96. r 


95 9 
95.8 

v • 


95.4 
'95.5 


95.6 
95.2* 


94.6 
94.1 




Non-f~F ^tiiHpnt in 

• ^V^ll 51UUCMI III 

~ Jitle 1 'school 


r\trdUI II V 

Math 


Q/l 7 
, 74/ 

94 7 


95.7 
95.6 


95 8* 
95.6 . 


95.9 
"95.8 


95.9 
"95.7 


m 

96 0 
96.0 


+ 


♦ 

C^on-CE student in 
Other-CE school 


Reading 
Math 


96 1 
96.0 


96.7 
96.6 


96.8 
f 96.7 - 


96.4 ' 
96.3 


96.6 
96.6 


-96.0 
96.0 




Non-CE student in • 
l^on-CE school 


Reading 
Math 


95 7* 
95.8 


96.4 
96.4 * 


^ 97.0 
97 0 


* 

96.6 
96.6 


96.5 
; 96.5 


* 96.1 
96.1 



- Note Standard deviations, sample siz£s, totals, and means calculated to two decimal places can be found 
r * 1 Table B of the Appendix _ ' 



m \ . * Table 5-3 

Percentage of Teachers in Title I Schools Who Reported That Non-Participants 
Are Involved in Alternative School Activities While Their CE Students 
Participate in CE Activities 



^Type of Activities in Which 
Non-Participants Are Involved * ' 

: 1 ? : — '• 


Reading Teachers for. 
Reading Activities 


Math Tochers for 
Math Activities 


Readings language arts activities 


89 2 


34.5 . 




Math activities 


* 29 8 






r Other subjecf matter activities 


42.8 






Study time 

<* 


.31.2 


30.2 * 




Student selected activities- 


27.4 . ' 


26.8 




Visits to the school library 


16.3 


10.2 




Physical education activities 


46 . < 


4.3 




Field trips 

r 1 


1.8 


- * 

0.9 





Note Numbers supporting the percentages, separately by grade, and other relevant statistics can be found in 
* Tables C and 0 of the Appendix.C 
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93 



We had data on CE participation for two years at the time Qongress requested the informa- 
tion (see Kenoyer, Cooper, Saxton, and Hoepfner, 1980 for a comprehensive discussion of 
the data) For the first year (1975-76) we merely had retrospective reports of whether the 
student participated in any CE reading or math programs. In the 1976-77 school year our 
information included whether the CE was Title I or some other kihd. We also collected the 
participation information from two sources, the primary source being a school coordinator 
who was to consult school or district records, and the secondary source being the 
classroom teachers who simply reported their knowledge (perceptions) about* each 
student's CE participation (see Hemenway, Wang., Kenoyer; Hoepfner„ Bear, and Smith, 

• for a discussion of the problems in obtaining this seemingly simple information). 

Tables 5-4 and 5-5 present information on CE continuation in two different ways. In Table 
5 4 we categorized students by their current CE statuses, and then tabulated how many 
participated in the previous year. Because- reading CE is more common than math, the 
percentages are alwa>s higher for reading. Among the "three, groups of CE participation, 
Title l^tudents have the highest rate of previous participation. (This finding could be the 

# result of numerous causes— the transftdry 'nature of many other-CE programs, the lack of 
success of Title I in effecting achievement growth that results in a student's disqualification, 
the fact that Title I participants are the lowest of the low achievers, etO-Ths, converse i$ 
presented in Table 5-5, where all the CE participants in the first y^ar are tabulated by their 
current-year status The percentages are highest for Title I, probably becausaitisjhe largest 
CE program. v . " ^ \ v 

We were also interested in learning the extent.of agreement between our two sources of 
information on CE participation, *and so calculated, the percentage of teachers' reports 
(secondary sources) that .agreed with'our primary sources. The agreement rates, presented 



Table 5-^ " 

Percentage of Students. Who ReceiverfxE In the Previous Year, 
. by Current CE Status and,Grade ^^^l 





CE Participation j * 




Gr. 1 


Gr* 2 


°Gr. 3 


Gr. 4 


Gr. 5 


Gr. 6 




Title 1 participants in 1976-77 


Reading 


23.3 


50,2 % 


62 3, 


64.1 


67,8 


67.2 * 


1 • 


vtybo received CE in i 975-76 


Math 


26.6 


38.6 


47 4 x 


47 0 


56.2 


58 3* 




t * 

Other-CE participants in 1976-77 


Reading 


276 


35,6 


a 

42 1 


'51,6 


46 6; 


47 0 




who received CE \y 1 975-76 


Math 


29.2 


33.6 


23 7 


34 7 


31.5* 


>30.9 




Non-CE students in 1976-77 


Reading 


3 3 


64 


' 9.4 


, 8.9 


e 






who received CE in 1975-76 


Math 


2.2 % 


' 2 7 


3 8 


4.5 




















Jt — 



Note Sample sues dn vvhich the 'percentages are based can be found in Table E in the Appendix. 
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Table 5-S" 

Percentage of Students Who Received CE in the Current Year, 
by CE Status in the Previous Year and Grade 





- 














CE Participation 




Gi\ 1 * 


Gr 2 






Gr. 5 


Gr. 6 


CE participants in 1975-76, who 
received Titlp Hn 1976*77 


Reading 
Math 


33.4 
27,2 


43 7 
351 


43 9 
41 7 


400 8 
33.4 


< 40.6 
36.2 ' 


35.3 
29.8 


CE^participants in 1975-76 who 
received Other CE in 1976-77 


Reading „ 
Math 


33,7 
39.2 


. 25.7 

3^7 

> 


21 4 
21 0' 


25.6 
28.1 


23.6 
22.6 


27.6 
25.0 


CE participants in 1975-76 who 
no longer received CE in 1976-77 


Reading 
Math 


32 9 

33 5 


30 s 7' 
32.1 . 


34 6 
37.3 


35.0 
38.5 


35.8 
■41 3 


37.1 
45.2 



Note Discrepances <*e due to rounding errors Sample sizes\>n which the percentages are based can be found 
in Table F of the Appendix 

» 

in Table 5-6. are surprisingly low, especially for othe; CE. We expected that other CE would 
have the lowest agreement, simply because 'the programs are generally not as highly 
regulated and controlled as Title I. These findings are cause for serious concernvhow can 
teachers effectively deliver supplementary CE services ta students if they don't kncV which 
students are participating? 



-»). - Table 5-6 , f 

I ' \ / 

Percentage of Teachers' Reports of CE Participation That 

Agree With SES CE Participation Status 



C£ Participation Status 


Cr.-I 


Gr. 2 


Gr. 3 


Gr. 4 


Gr. 5 


Gr. 6 










Reading 






Title 1 ' * 


62.1 


78 5 


77 7 


79.8 


80.5 


79,4 


Other-CE s 


41.0 


. * 47.1 


43 0 


47.7 


49.3 


"47.3 


NoCE 


'95 0 


94.1 


94.2 


94,3 


94 8 


94.6 










Math 






Title 1 


58 5 


69.8 


. 70.0 


7fl 


77 3 * 


74.1 " 


Other-CE 


36 8 


*34 1 


29.3 


35,2" 


34.9 


32.2 


"NoCE 


* 96 7 


96 6 


96.9 


96 4 


96.2 . 


97.0 



Note Numbers on which the percentages are based and explanations of the data sources can be found in Table 
G of the Appendix « 
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Length of Schoof Year, by Re/ding and Math CE Participation and Grade 


* 










♦ 

>* 






• 










t 


















♦ 


i 




In* Title I- Schools 












In Schools with Only Other CE 








« 

In Schools with 
No CE 


Grade 


Title 1 Student's 


Other-CE Students 


Non-CE Students 


Other-CE Students 


1 ' : 

Non-CE Students 


All Students 


Mean • 


S.D. 
r 




Mean 


s.p. 


N 


Mean 


S.D. 


N 


Mean 


S.D. 


, N 


Mean 


S.D. 


N 


Mean 


S.D. 


N 




















Reading 












• 








1 


176 71 


4 80 


1,&2 


1 76 77 


'*4 73 


815 


1 76 21 


© J J 


• 

*6,61 3 


1 78 58 


3 50 


533 




5 00 


2,786 


1 77 02f 


** 5 33 


1,268 




175 77 


6 48 


1 ,762 


1 78 02 


2 98 * 


8,26 


1 76 96, 


5 62 


5 462 


177 70 


3 92 


588 


1 78 34 


4 8o 


2,441 
i 


177 26 


5 10 


1,111 


3 


1 76 84 


5 47 


1 880 


1 78 33 

r 


2 91 


* 706 
682 


177 48 


j U5 


5,278 


1 77 97 


4 16 


558 


1 78 38 


4 87 


2,489 


1 77 15 


4 % 


1,160 


» 4 ' 


* ' 1 77 78 


5 46 


I 540 


1 77 94 


3 06 


1 77 37 


4 93 


5,434 


176 83 


3 43 


527 


1 7764 


"4 14^ 


2,487 


175 36 


6 88 


1,184 


5 


178 24 


5 41 


1 423 


* 1 77 98 


3 3J 


640 


177 83 


5 14 


5,622 


177 41 


4 10. 


542 


177 87 


3% 


2,640 , 


175^59 


6 73 


1,250 


6 


177 51 ' 

V 


5 52 , 


1 163 


k 177 08 


, 3 58 


724 s 


176 94 


5 18 


5,962 


176% 
< 


3 6Gt 


541 


177 57 


3 54 


4,334 


175 84 


6 03 


1,662 


Total 


• 177 07 


5 62 • 


9 370 


V 17766* 


3 56 


i 

4,393 


177 10 


550 


34,371 


17? 58 
> 


3 85 


3,289 


177 94 


4 36 


17,177 


* 1 76 33 


5% 


7,635 
















• 




* Math 




















r * 


177 04 


4 45 


850 


176 38 


4 45 


731 


176.26 


6 41 


7411 


179 00 


3 74 


361 


178" 09 


4*90 


2,957 


177 02 


5 34 


1,264 


2 


J 7 f V 


690 \ 


858 


177 06 


309 


626 


177 0* 


5 59 


6 $45 


178 34 


3 10 


306 


1 78 20 


484 • 


2,716* ' 


177 30 


5 07 


1,104 " 




1 76 63* 


5 15 


1 006 


• 177 16 


3 23 


532 


177 55 


■ 509 


■ 6,322 


*178 46 


3,56 


.389 


178 28 


4 90 v 


2,657 


177 15 


4 97 


1,158 


• 


« 177 72 . 


^5*19 


895 


177 14 


3 5V 


' 626 
w 


177 49 


499 


6,124 


177 21 


2 63 


362 


177 53 


4#? 


2,646 


175 35 


688 


N,I83 


' 5 


177 38 


6 23 


819 


. 177 11 


3 77 


• 559 


178 05 


.4 99 


6,326 


177,21*. 




404 


177 88 


4 02 


2,773 


•175 59 


6 74 


1,247. 


* 6 


* 1 75 89 


580 


657 


M77 35 » 


W 


634 
1 


177 11 


.5 16 


6,546 


177 17 


2 82 « 


' 424* 


J77 54 


3 61 


4,440 


175 84* 


6 03 


K659 


Tfltal 


176 57* 


5 74 


5,085 


1 77 o't * 


63 


-J 

3,688 ^ 






39,274 - 


177 86 


3 38 


• 2,246 . 


. J77 89 




4 38 


1*189 


176 33 


J 

5% 


7,615 



37 



r 

Table B 

t . * 

i 

Rates of Attendance, by Reading and Math CE Participation and Grade 















































In Title 1 Schools 












In Schools with Only Other CE 






( 


In Schools with 




































NoCE 




Grade 


THk 1 Students 


Other-CE Students 


Non-CE Students 


Other-CE Studeqtsj 


* Non-f E Students 




All Students 




Mean 


3. LA 




Mean 


S.D. 


N 


Mean 


S.D. 


N 


Mean 


S.D. 


N 


Mean 


( S.D. 


N 


Mean N ^S.D. 


N 




















Reading 














i 






• 

1 


92 95 


7 98 


1.572 


93 78* 


7 38 


7% 


. 9472 


6.81 


6,252 


94 59 


644 


521 


96 08 


4.95 


2.685 


95 74 


6.15 


1,201 


^ 2 


94 55 


6 65 


1,741 


94 63 


6 32 


810 


95 66 


5 77 


5,166 


95 82 


f. 

5 63 


575 


96 69 m 


4 26 


2,341 


% 36 


5 02 


1,056 


3 


94 11 


7 14 


1.847 


94 71 


6-66 


689 


95 80 


5 95 


4,997 


95 90 


5 11 


530- 


% 76 


4 57 


2,446* 


*96 96 


4.64 


1,106 


4 


94 56 " 


7*8 


1.524 


95 23 


6 37 


675 


95 91 


6 42 


5,172 


p 95 35 


5 82 


520 


_ 96 41 


j>07 


2,414 




4 68 


1,150 


* 5 


94 41 


7 91 • 


1,403 


94 14 


7 83 


623 


95 86. 


6 55 


5,351 


95 56 


5 79 


540 


96 63 ' 


4 85* 


2,562 


% 53 


,5 05 


1,224 


6 


- 94 89 


7 22 \ 


1 146 


3 


/ Uj 


715 


9604 


f 6 09 


5,769 


94 61 


664 


539 


96 02 


5 74 


4,259 * 

> 


96 05 


5 78 


r 

W>632 


Total 


94 21 


7 38 


9,233 


94 57 






95 64 


6 31 


32,707 


95 31 


5 94 


3.225 


96 38 


5 03 


^ 16,707 

V 


% 35 


5 31 


7,369 










' 










Math 










- 












































& 




1 


91 57 


891 
• 


829 


94 10 


699 




94 70 




7,061 


94.93 


6 30 


349 


95% 


508 


2,857 , 


95 78 


s 608 


1,198 


» 2 


93 52 


7 04 


842 


*95 02 


5 77 ' 


6fl 


95 59 


584 


6,254 


% 13* 


5 59 


294 


% 56 


4 45 


2,619 


% 40 


499 


1,051 


3 


9} 77 


699 


978 


94 30 


650 


523 


95 62 


/ 6 18 


6,031 


95 80 


5 27 


362 


% 72 


4 58 , 


«2,614 


% 95 


4.66 


1,106 


' 4 


94 26 


7 35' 


'887 * 


95 09 


6 28 


[ 620 


' %81 . 


6 54 


5,856 


95 47 


5 34 


356 


96 33 


t 5 20 


2,572 


96 62 


468 


1,150 


5 


0 94 16 


8 78 


816 


94 15 ' 


800 


521 


95 74^ 


6 48 


6,038 


95 24 


609 


403 


96 62 




2,697 


'96 54 


5 05 


1,222 


6 


94 10 


7 75 


651 


95 55 * 


681 


625 


96 01 


6 03 


6.347 


* 94 08 


7 63 


422 


96 04 e 


c 

5 61 


_ 4,371 


96 06 


5 77 


1,631/ 


Total 


93 56 


786 


5.003 


94 71 


6 76 


3.6 # 1 0 


95 56 


6 33 


37,5 V 


95 22 


6 18 

«> 


2086 


96 33 


5 05 


17,730 


% 36 


*5 29 


7,358 








* 




* 




• ■> 
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* • 
















98 

























































Table C 



Percentage of Reading Teachers in Title I Schools Who Reported That Non-Participants Are" Involved in 
Alternative School Activities While Their CE Students Participate in Compensatory Reading Activities 



Type of Activities 9 
in Which Non- 
Participants are s 
Involved** 


Grade 1 * 
(N*272) 


Grade 2 
(N-274) 


Grade 3 
(N* 268) 


Grade 4 
(N « 230) 


Grade 5 

k / Kl Tni\ " 


" Gracte 6 
(tt-1701, 


t 

*Total 

(N« 1,4^5) . 


% 


Count* 


% 


Count 

V 


; * 

% Count 


% 


Count 


*> Lount 


v or 


Count 


% Count 


Other reading 1 * 
or language , 
arts activities 


# 96 0 


261 


89.8 


246 


89.6 240 


80,9 


186 


84.1 ,169 


77 7 
» 


132 


\ 

07 1 1 ,c 


* Other reading * 
or language arts 
activities only 


36.8 


V 

100 


30 7 




29.9 80 


24 8 


57 


,35 3 71 


) 

29,4 


j\j 


.31.2 442 


Math activities 


34.9 


95 


32.5 


09* 


28.7 ' 77 


30,9 


*71 


23.4. 47 


24 7 


42 


29.8 421 


Activities related * 
to other subjec| 
matter areas 


29 8 


81 


40.5 


111* 


43 3 11^ 


49.6 


114 


48.3 97 


51 2 


87 


42 8 % 606 


Study time 


22 1 


60 , 


29.6 


81 


34-. 1 92 


36 5 


84 


» 31 3 63 


35,9 


61 


' 31.2' 441 


Student selected * 
activities 


^34.9 


95 


30.7 


84 


29.9 -80 


21 7 


50 


21.9 44 


20.6 


35 


27 4 388 


Visits \6 the 
school library * 


14.0' 


38 


13.5 


• 37 


* t 

14.9 * 40 


18.7 


43 


20.4 41 


18.2 


31 .« 


16*3 230 


Physical evocation ^ 
activities v < % 


2.6 


■ 7. 


5 5 


15 


5.2' - 14^1 6.1 


14 


3 5, 1^ 




8 


-4.6 65 


Field'tnps 

* 


. 1.1 


'3 


3.3 


' 9 - 


1.1 3 


0.4 


1 


1.5 3* 


3.5 


6 


' W 25 ' 



ERLC 



Table C (Continued) 



f 

Reading teachers whW 
have CE students 


100.0 


291 


i 

k 

1000 


289 


100.0 


282 


Response 

_ 100.0 


-j — 

Pattern 

245 


100.0 


2J9 


100.0 


182 


■100.0 


1,498 


Who marked one or - 






























more of the above * 
activities 


93.5 


272 


94.8 


274 ' 


95.0 
•«> . 


°268 


93.9 


230 


%.2 


201 


93.4 


170 


94.5 

u 


1,415 


of the above* 
activities 


6.5 


19 




15 


5.0 

n 


14 


6.1 


15 




8 


6.6' 


12 


5.5 


8'3 



Qon-participants. 
-'Multiple responses Were allowed. 



one alternative" activity for 



Table D 1 



Percentage of Math Teachers in Title I Schools Who Reported That Non-Participants Are Involved in 
Alternative School Activities While Their CE Students Participate in Compensatory Reading Activities 



s * 

- Type of Activities 
in Which Non- 
Participants are 
Involved** 

• 


r 

. Grade 1 
* tN-137) 


Grade 2 
(N-121) 


Graded 
(N-136) 


* 6*ai 

w- 


4e4 * 

US) 


Grade 5 v 
(N-118) 


v Grade 6 
.(N-106) 


Grade 7 
(N*744) 


% 


Count* 


<*> 


Count ' 


% 


Count 


t< 9 
% 


Count 


% Count 


% Count / 


% Count 


Other math , . 
. activities 


.81^0 




77.7 


94 


83.1 


113 


"i 

80.2 


101 


83. U 98 


86.8J 92 


81.9 609 


Other math * 
activttierfonly 


36.5 


50 


3fc,4 


38^ 


27.2 


37 


36.5 


46 


34.7 4f 


••r * ^ 

^,47.2. 5Q ' 


35.2 262 


Reading or 
other language 
arts activities « 


366 


% 

50 
v 


45.5 


. 55 
« 


39.0 




29.4 


37 


28.0 13 


27.4 29 


34.5- 257 


— « : — ' 



ERJC f , . 



lull 













» 








V 








\ 










- 






* 
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Table D (Continued) 




• 






I 


Activities related 




* 




















• 


to other subject 
matterareas 


26*3 


' 36 


31.4 


38 


36.8 50 


38.1 


48 


40.7 48 


38.7 41 


35.1 


f 




Study time 


22 6 


31 


24.0 


'29 


'34 6 47 


34 1 


43 


34.8 *41 


32.1 " 34 


30.2 


225 




Student selected 
activities , 


29.9^ 


41 


27 3 


/ 

3-3 


34.6 47 


27.8 


35 


i 

21.2 25 


17.0 r , 18 


56.8 


199 




Visits to the 
school library 


44 


6 


7.4 


9 


U 8 16 


12.7 


16 < 


' 14.4 17 


113 12 


10.2 N 


76 




Physical education 
activities 


h 


2 
r 


4.1 


5 


7.4 10 




8 


.4.2 5 , 


19 2 - 


4.3 


32 




Field Trips 


0.7 < 


1 


t 

2 5 


_ 3 


• 

0.7, 1 


OS 


1 


09 k , 1 


0.0 * 0 


0.9 












- 


« 




Response Pattern 












.Math teachers who 
Ahave CE students 


1000 


176 


100.0 


156 


i 100.0 170 


ioo:o . 


167 


J 00.0 * 141 


106.0 125 


100.0^ 


935 




Who marked one or 


















• 


- 




• 


more of the above 
•activities * 


/77 8 


137 


77 6 


121 ' 


80.0 1 36 


75.4 


126 


83.7 118 


' 84.8 -1106 


79.6 


7 A A 




• Who marked none 






• 




















of the above 
activities 


22.2 


39 


22.4 * 


35 


20.0 34 


24.6 / 


41 


16.3 .23 


15.2 19 - 
* 


20.4 


191 




/Based on math teachers who have CE students in Title 1 schools and who have reported^Tleas! one alternative activity for 
non-participants • * \ ^ 


• 






**Multiple-r&pon*se$ wire allowed. 


































♦ i 






* 
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Students' Receipt of CEIn 1975-76, by Their CE Participation Status in 1976-77' 



CE Participation- 
Statu s^n " 
1976-77 
V 



0 

Receipt 
ofCE in 
1975-/ 



Grade 1 



Grade 2* 



Grade 3 



Grade 4 



Grade 5 



Number Percent J ' Number % Percent ' Number Percent Number Percent 



7^ 



Grade 6 



Number Percent 



Number Percenf 



'■■) 



\ 



Reading 



V Title I % 
students 
& 

Jpther-CE 
Students 



Non-CE 
students 



1 Received 

Not Received 

Total 
n 

, Received 
Not Recerved' 
Total 

Received / 
NqI Received 
Total 



350 


23 3 




788 


502 


1 152 # 


76 7 




783 


49 8 


1,502 


100,0 




) 571 


100 0 




27 6 




463 


35 8 


9,26 


- 72 4 




*30 




1,279 


1000 




1 293 "* * 


100TJ 


345 • 


, 33 




553 


64 


10 16J 


• % 7 




8 147* 


93 6 


* 10,508, 


1000 




8,700 


1000 



1,045 

632 , 
1,677 

5 TO 

I Uf2 

823 
7,934" 
.8 757 



62 3 
. 37 * 
' 100 0 

42 1 
.57 9 

iooi) 

94 
' 906 
100 0 



8% 
502 
1 398 


64 1 

35 9 
1000 


. ' 888 
422 
1 310 


67 8 
32 2 
1000* 


726^ 
355 
1,081 


67 2 
32 8 
1000 


560 
5^8 
4,098 » 


51 0 
49 0 
1000 


5* 

591 
1,106 


46 6' ' 
53 4 
1000 


566 ' 
639 
4 1,205 


47 0 
53 0 
100 0 


783 
8,039 
8 822 


,8 9- 
91 1 
100 0 


783 
8,427 
9,210 


" 85* 
91 5 
1000 


762 
* 10,944 
-11,706 


• 

6 5 
93 5' 
100 0 



Total Number of Students* 



13,289 



11 564 



11 646 



11,318 



11,626 



13,992 



I 

Title I 
> students 



Other-CE 
students 



Non-CE 
students 



Math 



Received 
Not Received 
Tbt.il 

Received 
Not Received 
Total 

Received 
Not Received 
Total 



209 

A 577 

.ijOl 
729 
1*030 



* 257 < 
11 207 
11 464 S 
1 



26 6 
71*4 
1000 

29X« 

70 8 
1000 

•97 8 
1000 



295 
/ 469 
764 ♦ 


38 6 
61 4 
100 0 r 


865 


47 4 
52 6 
1000 


367. 
414 
, 781 


47 0 

53 0 , 
1000 


422 
329 
751 


56 2 
43 8 
1000 


357 
255 
612 


58 3 
41 7 
* 100 0 


m 275 
544 
819 


33 6 

66V » 
"TD00 


206 * 
663 
, 869 


23 7 
76 3 
TOO 0 


309 
581 ' 
890 


34 7 
* 65 3 
J 00 0 


, 263 
571 
834^ 


31 5 
685 
1000 


299 
669 
968 


309, 
69 1 
100 0 


270 
-9,657 
9,927 


2 7 
97 3 
* ,4000 


. 367 
9,411 
9 778 


38 
. 962 
1000 


424 
9,067 k 
9 491 


45 * 

95 5 

100 0 ' 


481 
* 9,406 
9,887 


49 
* 95 1 
1000 * 


542 

11,760 
12.3*02 


4 4 
% 95 6 
100 0 



Total Number of Students* 



13 280 



11,510 



11,512 



11,162 




1 13,882 



• 'Receipt ot CE in 1975-76 was based on teacher's report in SBC CE participation status in 1976-77 was based on CEft ' ' ' 

''^ntSKT r rdS aVaHab,e hT m '™ eXC,U ' ded n0t ^ Cr>6e 1, CE rece.pt ■„ 1975-76 referred to their V 
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Table F 




i 

' i » 


■ 




' c 


E^articipation Status for Student/who "Received CE in 1975=76 


* \ 




Grade 




Title 1 
Students 


Other*CE 
Students 


• 


Non-CE 
' Students ' 






s 






. r 


Reading ' 










\ 


Number 
f Percent 


350 
33.4 


353 
33.7 




' 32.9 


1 AvtQ 

100.0 






2 


Number ▼ 
Percent » 


788 
43 ^ 


463 > 
25*7 




■J J J ^ 

30.7 


1 AAA ' 

100.1" 






3 


Nurpber 
Percent . 


1,045 
43.9 

< 


510 
21.4 




• 823 
34.6 


2,378 
99.9* • 






4 


Number 
• percent 


896 
40.0 * 


560* 
25.0 


V 


783 
35.0 


2,239 
100.0 






4 

5 ^* 


Nuxpber 
Percent 


888 
40.6 


*23.6 . 




783 " 
35.8 


2,186 
100.0 






*6 


Number 
Percent 


726 
35.3 


566 

• '27.6 




762 J 
37.1 


"2,054 
100.0 








i 




# Math 










t ' 

1 


t, 

. ^ 1 


Number 
Percent 


209 
27.2 


301 
39.2 


257 

33J 


7A7 
/O/ 

99.9"* 






2 


Number 
y Percent 


295 
.35.1 


275 
* " 32.7 




97H 

32.1 . ^ 


. 99.9" 






• * - 


Number 
* Percent 


4J0 
41.7 


206 
2L0 


i 


367 
37.3 


, 98? 

ioo.o 




* 


4 

V \ 


Number 
Percent 


367 
33.4 


309, 
/ 28.1 




^ 424 
38.5 


1,100 
100.0 






1 * 

5 


Number 
Percent 


422 
36.2 ~ 


1 0 263 ' 
. ' 22.6 


> 


40 1 

\ 41.3 ^ ' 


1,166, 

ido.i *• 








Number 
Percent ^ 


357 
• 29.8 


299 
, 25.0 % 


• 


*542 

45.2 * ' 


1,198 
100.0 






•Receipt of CE in 1975-7£ is-bas^d on teachers' report in SBC. ' 






• * 






A^t participation status in 1976-77 is 


bastion CER.. .*r-~ 








♦•Discrepancies'due to rounding errors. 




r 

t 


* 










104 

74 * ~ 






# 


















Table G 

Students' CE Participation Status in 1976-77 and 
Teachers' Reports of Their CE Receipt in 1976-77* 



Title I 
Students 



Other-CE 
Students y 



Noo-CE 
Students 



Total, 







Number 


Percent * 


Number 


Percent 


Number 


Percent 


Number 


Percent * 


> 

f 

. 1 

. \ 


Receiving 
Not Receiving 
Total 


1,045 
638 
1,683 


75 
46 
12 1 


Reading 

570 
819 
1,36^ 


• 

4 1 

5.9 
IjLO 


540 
10,335 
10,875 


39 
74 J. 
78.0 


** 

2,155. ' 
11,792 


84 6 ■ 
100 1 


2 


Receiving 
Total. 


1,418 


11.4 
3 1 
146 


&2 
766 
1,448 


5 5 

6 2 
11 7 


538 * 
8,601 
9,139 


4.3 
69 4 

73 7 


2.638 
9,755 
. 12,353 


*> 

21 3 
78 7 
1000 


3 
.) 


Receiving 
Not Receiving 
.Total 


1,480 
425 
1,905 


I 

t)2 0 
154. 


' 557, 
737 
1,294 


4 5 
60 
10 5 


533 
JB,625 
9,158 


43 
69 8 
74 1 " 


2,570 
9,787 
12,357 


20 8* 
79 2 
1000 


r 

4 


Receiving 
Not Receiving 
Total 


1,233 
312 
• 1,545 


10 4s 
26 
130 


572 
628* 
1,200 


48 

5 3 
10.1. 


526 
8,637 • 
9,163 

I 


44 

72.5 C 
77J0 


2,331 
9,577 . 
11.908 


19 6 
804 
1000 


5 


Receiving* 
Not Receiving 
Total 


, 1,177 
286 
1,463 


96 
2.3 
120 

> 


588 
605 
1,193 


4.8 
50 
9.8 


. ^ 493 
9,079 
9,572 


40 ' 
74 3 
. 783 


2.258 
9.970 
12,228 


18.5 
81 6 
10Q0 


6 


Receiving 
Not Receiving 
Total • 


931 
242 
1,173 


64 
1 7 
8 1 


'599 
668 
1,267 


42 • 

46 

88 


11,758 
12,009 


* 78.6' 
81 1 


2,181' 
12,268 
14,4^9 


15 1 
84.9 - 
1000 








♦ 


Math 












i 

I 


Receiving 
Not Receiving » 
Total* 


515 
366 
881 


3'7 , 

26 

63 


409 
703 
1,112 


29 
• 5.1 " 


391 

J 1,508 * 
11,899 


2.8 
82 8 
85 7 


, 1,315 
12,577 
13,892 , 


' 95 * 
905 
100.O 




Receiving 

Total 

\ 


600 
zou 

OOU 


* 49 
i. i 
7 0 


' ' 316 
a 610 
926 


2.6 
5 0 
7 6 


352 
10, 1 13 
10,465 


2,9 
82.6 
85 4 


1,268 . 
10,983 ' 
12,251 


10.4 : 
897 
10(h1 


3 


Receiving 
Not Receiving 
Total 


697 
300 

997 , 


57 

'2 5 . 
82 


' 274 
661 * 
935 


2 2 
5 4 

7 7 * 


322 
9,554 
10,276 


26 
81.5 
842 


z,293 
10,915 
^ 12,208 

1,322 
10,439 * 
11,7^1 


106 
89,4 
1000 




Receiving 
Not Receiving 
Total 


626 
255 ' 
881 


53 
22 

75 • 


339 
625 
964 


29 
5.3 
82^ 


357 
9,559 
9,916 


30 
81 3 
84.3 


11 2 
5 88.8 
1000 


5 


Receiving 
Not Receiving 
Total 


651 *■» 

191 

842 


54 
I 6 
70 


328 
611 
939 


2 7 
50 
7 8 


393 
9,945 
10,340 


\ 

33 
82 1 
85 3 


1,374 
10,747 
12,121 


11.3* 
887 
1000 


6 


Receiving 
Not Receiving 
Total 


488 
* 171 
' 659 


3 4 
1 2 
46 


334 
703 * 
1,037 * 


2 3 
49 

73 


377 
12,229 
12,606 


2,6 
85 5 
88 1 


' 1,199 
13,103 
14,302 


84 
9J6 

1000 . 
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CHAPTER 6. TITLE I STUDENTS AND SPECIAL SERVICES 



t . , ^ Judith A. Hemenway and Ralphj^oepfner^ . 

'In order to determine if the sample of Title I students m the Sustaining Effects 
Study (SES) contains a disproportional number of handicapped students, who 
might be expected to hold down achievement growth and thereby obscure * 
measures of program effectiveness, tabulations of student characteristics were 
* projected to^he nation. The projections indicate that the Title I sample does not ,\ 
have a disproportional share of students with handicappihg conditions. 

Categorical government programs have usually been designed to serve specific target 
populations as focused responses to. perceived needs. For Title I of ESEA, the ta r^P°P u ' a - 
tion i| cQmposed of low-achieving students in low-income neighborhoods. For PT* P 94-142 
(Education for All Handicapped Children Act), the targets are students with handicaps that 
impede learning or participation in school ^activities. Although the words in the guidelines 
for each program are clear, the differences in the concepts defining the two target 
populations are not so clear. In other words, it is the cause? of the low achievement that 
theoretically determines whether a*low-achieving child is a Title I student (arl educational 
or social caus^) or a 94-142 student (medical <\j>psychologicaI catfte). Because the cause is 
not always clear, especially when handicapping condition is ^ebatably medical (e.g., 
learning disabilities, mild educable rnental retardation, emotiOpal disturbance), ' we can 
expect some students to be in the wrong' program, some in both programs,' or some who, 
multiply eligible, are disqualified from each program on the basis .of an assumed qualifica- 
tion for the other. Birman (1979) studied the incidences of these occurrences and con- 
cluded *that duplication of services was not a major problem because teachers -and 
administrators made sure that service^ to multiply-eligible students were coordinated. 
Birman was' concerned with the duplication or denial of, services, and concerned also 
to enlighten the processes of making law and regulations so that such probrems wpuld 
be /resolve d„> * 

Complementary to Birman's concerns, interest in the issi>e of multiply-eligible students and 
their services in the SES was not directed at duplication q/ services, but at multiple par- 
ticipation. Receipt of Birman's report in Washington caused questions regarding just how 
prevalent it was that Title I participants were also handicapped. The concern for the* SES- 
was that if special-education children, who might Tiave low probabilities of normal achieve- 
ment growth, are counted as Title I students, they may depress the measures of growth for 
the Title I samples. If such;students are represented more in the Title I sample than in any 
comparison sample, and if fheir growth rates vn achievement cannot be expected to be the 
~ same as that for other children in the Title I sample, then compari?ons between samples 
may unfairly reflect poorty on the Title I program. ^ 

1F3?Ormation on students with problems^ 

An ^instrument in the study requested information relative to the issue of the incidence of 
^handkapp.edLiludem Title I j)articipants. Specifically, one item of the Student 
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Background Checklist asks for .teachers' observations and judgments v\he*fter the * student 
has physical, psychological, or adjostment % prot5tems that ^interfere with academic perfor- 
mance " It should be noted that the item posed no requirement for formal diagnosis by a 
physician, psychologist, sp^ch therapist, etc, the responses 'were teachers' observations 

* and judgments, and can be expected to reflect their attitudes and rationalizations to some 
unknown extent. National projections of the responses to this item were reported by 
Hemenway, Wang, Kenoyer, Hoepfner, Bear, and Smith (1978) in Table 4-7. Nationally, 11 
percent of the students were judged by their teachers as having such problems. The high 
value of this percentage makes us wary of assuming thatall, or even most, of these students 
are special, handicapped, or would qualify for services under P.L. 94-142. 

• • 
^ From responses t<^a second Jtem, we teamed that of the 1 1 percent, 4,8 percent received 
treatment provided by the school, 1.7 percent rec^ved some treatment provided by the 
school but still had at leasfone untreated problem, and 4.5 percent received no treatment 

* from the school It should also be noted that there ^s no specification that the treatment pro- 
vided by the schools met any current therapeutic standards. ' , i 

In Table 6-1, we present a nationally projected cross-tabulation of the responses to this item 
with the teachers' judgments of each studerlt's need for compensatory education (CELThe 



Table 6-1 



Teacher Judgments of Student Need for Compensatory Education 
and of Handicapping Conditions : , 
(Entries are percentages of students, nationwide.) 





Grade 


Teacher Judgment Student 
of Student Neeo* ' Has No 
forCE ' Problems 


School 
Gives 
Treatment 


Some * * 
Treatment, 
Problems Remain 


No 
Treatment 






Does not need CE 95 
* Needs CE / 72 


. 3 

12 


0 

V 

4 


2 

< 12 




2 


Does not need CE , 95 
Needs CE ' - * 76 


2 

10 


'i 
4 


2 

* 10 






Does'not need CE 95 
4 Needs CE ^ 77 ' 


2 
11 


■ • i . ; 


2' 
8 




4 


\ 

Does not need CE 94 \ 
Needs CE ^ 75 


* 1 
11 


• i 

4 


- • 2 
10 




5 


Does not need CE 96 
Needs tE ' 77 


1 

9 , 


1 * 

6 


2 

' 8 ' 




6 r. 
i 


Does ndt need CE " 96 * 
Needs CE % —79 


^ 1 . - 
10 


1 • *. • . 
4 


2 
7. 
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.data mTable.6-1 are useful, primarily as a validity check on the teachers' responses insofar 
as students with problems need extra services. In summary^ about 95 percent of the' 
students who 'do not need CE' have no problems that interfere with academic performance 
(the remaining 5 percent, presumably have problems that do not interfere so much as to 
make them need CE services). More critically, about 24 percent of, the students who 'need 
CE' are judged as having problems that interfere with academic performances (the remain- 
ing 76 percent presumably need it for other reasons'). We can see from Table 6-1 that . 
problems are much more common among the CE-needy student's. 

TITLE I STUDENTS AND ADDITIONAL CE SERVICES 

Because special services to children wfth handicappfng problems are frequently considered 
as compensatory services when the services are part of some program other than Title^we 
can, to some degree, ga^jge how large the problem of the contamination of the Trfle I 
sample with special students might be by determining how many Title I students also par-, 
ticipate in other CE programs Table 6-2 provides national percentages of students by grade 
and participation in CE' programs*. 

From Table 6-2 we can see that the percentages of children feceivmgmore than Title I ser- 
vices is very small. This finding is similar, in some respects, to that of Birman. 

In the lower half of Table 6-2, the children receiving CE services in addition to Title I are ' 
tabulated as percentages of the Title I population. These figures indicate that a substantial 
proportion of Title I students (about 1 ^percent) also receive other CE services. We interpret 



Table 6-2 



Percentage* of Students in. Title I Solely and With Other CE Services 



Students in Title,! Schools With: 


Gr. 1 


Cr. 2 


Gr. 3* 


4 Gr 4 4 


Gr. 5 


Gr. 6 




National Percentages 




p 






*j Reading — -* 










\ 




Title 1 an^ Other CE 


, 24 


2.7 


2 7 


1 6 


1.4 1 


0.8 


Title 1 only 


10.4 


' x 13.7 - 


13.0 


13.0 


10.8 


8.7. 


Other CE ogly , * 


. 6'4 


7 1 


59 


63 




5.7 


Math 














Title 1 and Other CE 




0 8 


'1 7 


1 3 


* 1.2 




Title 1 only 


4 5 5 


7.3 


66 


* 

6-8 


67 " 


J\ 


Other.CE only 


7.2 


6.9 


5 4 


66 


5.5 


4.8 ^ 


* • * 

Reading 
Math 

• 
* 


Percentages ofvAll Tiile 1 Students 

188* 165 172 
.14.1 9 9 20*5 


11 0 
16t\ 


15.2 


8.4 
12 7 1 
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these figures as extreme overestimates of the possible incidence of special students in TitleJ 
programs, because not all the additional CE services are of a special education nature. If the 
percentages were calculatecJ-on the basis of only those Title I students who are enrolled in 
schools that also participate in some othej CE, so that it would be possible for them to be 
. double pahicipant^.the projected percentages could be expected* to' be somewhat larger. s 

INCIDENCE OF CHILDREN IN EACH CE CATEGORY JUDGED TO HAVE PROBtEMS 

We can look more specifically at chilbs^njvith problems and their frequencies in different 
CE programs. A first look at whether the incidence of problem students in Title I programs is 
greater than, equal to, or less than tTie incidences in other-CE categories that migfit servk as • 
comparison groups 1Y1 the SES is provided in Table 6-3. The percentages of children judged 
by their teachers as having probjerm that interfere with academic performance are pro- 
jected to the national population for each of the categories<of CE participation. 

t From Table 6-3 we can see that the incidence of^eachers' judgments of having problems is 
lowest "for the Non-CE students, for both readifig and for math ^Considering reading CE par- 

, ticipants only, the incidence of judged problems i£ highest for other-CE students in ojher-CE 
schools ia possible reflection of the special nature of those CE programs) and for the doubly 
.served v students in Title I * 

Among the math students, the other-CE, students in other-CE schools have the highest in- 
cidence of judged problems, exceptor Grade 5, where the doubly served Title I students 
have a high incidence - Overall, however, judgments of problems are not notably more 
common among the doubly-served Title I -students. Such judgments are similar for almost 
•all CE categories and grades Therefore one can conclude that later comparisions of Title I . 
students to other CE students will not be greatly influenced by differential incidences of 
problems. " * 

• INCIDENCE OF CHILDREN IN EACH CE CATEGORY WHO RECEIVE TREATMENT FOR 
THEIR PROBLEMS 

Looking more closely at the children judged by 'their teachers to have problems that 

* interfere \vith academic achievement, Table'6-4 presents percentages of tho^e students - 
who received treatments 

For grades I and 2, the doubly served Title ,1'students have a higher incidence of treatment 
than any other^ category, but this fs not the case m the higher grades, where other-CE 
students ffequertfjy have the highest incidence of treatment. 

CONCLUSIONS!^ . 

Teachers' judgments of their students' physical/psychological, or adjustment proBlems that 
interfere with academic performances supported to some degree by their judgments of 
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those students' CE needs. The number of Title I students who are also served by additional 
CE programs is about 14 percent. 'Those doubly served students frequently, but not always, 
have^a higher incidence of judged problems, however, the incidence is not, very dissimilar 
from tha.t for the other categories of CE. Among the students with-judged problems, the 
doubly served have a slightly higher incidence of receiiing treatment frbm the school, but 
only at grades ,1 and 2 The data presented do not argue? except very weakly, that the Title I 
program might siWer in comparison to other programs because of the inordinate incidence 
of handicapped students. > ' • • V 



Table 6-3 



/ 



Percentages of Students Judged To Have Problems That 

Interfere with Academic Achievement, Nationwide . - . \ff ' ' 



CE Participation 


Gr. 1 


Gr. 2 


Gr. 3 


Gr. 4 


* Gr. 5 


Gr. 6 




Reading 












V 

Students in Title 1 Vhnnk 














Title 1 and other CE 


18 2 


18 2 ~ 


162 


31 % 


35. o 


29 7 


. Title 1 only 


20 5 v 


22 ] 


176 


18 3 


17 3 


17 1 


vainer l t only ♦ 


13 7 


20 5 


180 


16 


18 1 


13 2 


No CE , 


10 6 


8 6 


7 Q 

/ j 






D 9 


9 

Students in Schools with Only Other CE 










■ V 




Other CE 


25 9 


25 0 


276 


30.2 


" 29 3 


26 3 


-Nort ) 


83 


6.2 


8^ 


10.1 


^48 


5 6 


Students in Schools with No CE 














All students 


11 4 


8.8 


100 


9 5 


-A- 


8 1 


< 


Math* 












Students in Title 1 Schools 














Title 1 and other CE 


21 8 


170 


164 


13 5 


37 5 - 


22 0 


T?tle 1 pn ly 


18 7 


19 1 


19 4 


18 0 


16.0 


12 9 


Other CE only 


•12 3 ' 


15.5 . 


14 1 


17 9 


\8 7 ■ 


13 3 


No CE 


11 8 


1 1 5 


96 


11 1 


Va i 


83 


Students in Schools with Only Other CE 


* 




* - 








Other CE 


24 8 


22 6 


23 6 


30 0 


27 7 


25 3 


y No CE 


9 8 , 


8.8 


10 4 


10 7 


6.0 , 


6 2 
« 


Students in Schools with No CE 


>, ' 












All students 


11 4 - 


8.8 


100 


9 5 




8 1 


















V 



Table 6-4 



" * • 

* * Percentages of Students Who Received Treatment for Their Problems,* 
by Grade and by CE Category (Based on Nationally Projected Counts) 



t CE Participation 


Giv1 


Cr. 2 


Cr. 3 


Cr. 4 


Cr. 5 


Gr.£ 


** • 


v£k *s #1 inn 






• 






Students in Title 1 Schools, 




• 










'-Title. 1 and other CE 


78.6 


70 3 


69.1 


58.7 


73.1 


65.3 


Title l only 




62 3 


55 1 


58 5 f 


60.1 


61.7 


. Other CE only • \, 


54 4 


46^' 


62.8 


• 69 8 ' 


62 5 


68.2 




• 53 7 


49 4 


59 5 


57.3 


57 9 • 


57.3 


Students in"Schoofs i with Only Other CE 














Other CE 


65 9^ 


68 6 


79.2 


80.8 


„' 70 6 „ 


89 0 


No CE •* « 


56.6 


58 1 


44 3 


57^4 - 


50 0 


**> 

50 0 


Students in Scnoois'with No CE c 














All students 


58 3 


44 3 


56 0 


- 50 0 


58 0 


52 4 




i 

Math 


* 






9 




Students in Title 1 Schools 












'Title 1 and other Cf 


• 73-9 


78 8 


78 2 


62.? 


61 3 


83.3 


Title 1 only * ' 


. 67 9 


.54.9 


45.4 


'72 6' 


46.9 


89.2 


Other CE only 




J 1 o 


f\7 A 


A^ ft 


7A 1 


by z 


No'CE 4 


^ V 54 6 


"54 7 


61 V 


59.4 


59.8 


53<0 * 


, Students in Schools with Only Othe/ CE ■ 














Other CE 


62.9 


59 5 


76 2 


87 3 


76.5 


89.8 * 


No CE 


' 59 1 * 


64 1 


'52.9 ' 


57.0 


50.8 


54.8 


Students in Schools with No CE 












/ 


All students . 


58 3 


44 3 


_ 56.0 


50.0 


58.0 


52.4 



treatment was defined for this table as full or partial treatment provided by the school.. 
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PART II. ACHIEVEMENT AND COMPENSATORY EDUCATION 



Many inquiries durtrfiflfee 1978 reauthorization (and annually during appropriations 
hearings) concerned hcJWand how well Title- 1 services (and CE in general) are distributed aj 
the student level ,ln the first chapter of this part, we develop severahndexes of school-level 
'targeting'— tjje extent to which low achievers are selected for CE participation. Some 
caution^in the interpretation of the figures is warranted, however, since the application of 
any'of ou\targeting indexes to all schools, without regard to the philosophies underlying 
their CE programs and the wide range of acceptable 'program structutfng, results in some 
unfair evaluations of schools. The indexes are compared on the^basis of their merits 
and shortcomings. 1 

\ 

The second chapter addresses the use of teachers' judgments- to select students for CE, a 
prattice that is widespread but not, well understood in terms of accuracy jn comparison to' 
scores on achievement tests. We wanted to demonstrate how much agreement tnere was, 
so that if agreement were ftigh the less costly . judgment could be encouraged as an 
acceptable method for selecting students. Judgments being what they are, we also studied 
wha&might influence them, so that, even if accuratg, systematic biases or errors could be* 
pointed out. * * 
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CHAPTER 7. TARGETING OF COMPENSATORY-EDUCATION 

SERVICES' 

Charles E. Kenoyer and Deborah M. Cooper • \ - 

. ; T • * >. • 9 * ! 

/n //>/s study we attempted to develop and test a method for calculating an 
' % /ncfex that would reflect how well compensatory education (CE) is being targeted 
to those stOdents who need iuSeveral definitions of participation and of need for 
CE are studied, reflecting different notions of ideal allocation? of <servfces. 
Twenty-five indexes are discussed and evaluated; none can accommodate all the 
acceptable and reasonable variations in how CE services are allocated to 
students. Ihstead of recommending a universal index, then, w x e urge that whhn a 
targeting index is needed it be selected on an ad hoc basis to correspond with the 
rationale and implications of the program's laws or regulations, and that the 
*be fully aware of its' limitations. • \ • 

A study of the targeting of compensatory education was conducted as part of the Sustaining 
Effects Study (SES) in an attempt to determine the degree to which CE services are targeted 
■ to needy students. The purpose of the study was to develop an index to indicate matches 
between the need for and participatiorr'in CE programs, and to examine the merits of the 
index. The findings are intended to inform* policy-makers and decision-makers regarding 
the usefulness of the derived targeting indexes/wflfch. might then be considered for use in 
studies of compliance. ■ " ' ' - " 

> • f * 

* This paper does not address CE targeting at high administrative levels, such as states ancf 
.districts/but focuses on how well CE is targeted to children in schools that receive CE funds. 
The^ targeting indexes we report, therefore, assiifhe all the prior allocations and are based 
on the within-school success in reaching student^ defined as needing extra services. A 

^.variety of indexes was developed, based on different concepts of what constitute accept- 
r able or ideal allocations of CE services. We examined such a v^iety for two reasons. First, 
'most CE programs (including Title I, which is the most carefully* regulated) allow for some 
, flexibility in how schools can Select students to receive services. A range of selection 
procedures is allowed to reach the goal of serving the* neediest students— narrowly 
constrained regulations will not generally work. Second, many'schools offer more than one 
CE program, each allowing different student-selection procedures, We study the targeting 
of CE services while taking into consideration variant selection procedures. In this manner 
we consider various ways to define student needs {test scores, teachers' judgments), jDar- 
ticipation (yes/no, or intensity of service), and CE goals (all students should be needy; all 
non-CE students should not) and develop .indexes based on them. % 

* We havelbken th§ position that no single index of targeting can serve for all situations in. 
which targeting is to be assessed/What constitutes proper assignment of CE to students may 
be specified in 1 various ways, depending on several considerations in addition to the regula- 
tions of the CE program. The selection of a targeting index for any' given evaluation 
problem, then, requires that the evaluator exercise judgmenfrFor this reason, with each 

-•1 ' ~V 



index we present the rationale for its derivation and some empirical data on its distribution 
in the SES sample. f t 

Due to the variety of situations, the evaluator must consider many things about the program 
before t housing a particular index to assess its targeting properties: 

• flow prescriptive are the regulations of the program in specifying which students are 
to participate? Can a school, under some circumstances, decide to select aH students? 
Can Jt decide to limit the grades (and thereby the students) in which the program is to 
be implemented? Can- it decide (o continue services to students even though they are 
technicaNy disqualified by* other Criteria? * 

• Should the" index be sensitive tfe\conditions in which funds are inrsufficient to meet 
^school needs? For example, an index that considers only how many of the students are 

needy may be insensitive to funding limitations. But an index lhat considers how 
many of tne needy students are tQ be selected fpr the program may be solely sensitive 
to the 'variability of funds. 

; ' / 4P 

> / % 

• Are the allocation procedures bafed-on rational normative data? 

• Should the targeting index consider only the selection of students, or should it con- 
sider the instructional services to be received by them? Selection is, in fact, little more 
than a designation that is part of the allocation procedures; it may not be a valid 
indicator of the receipt of any slices likely to called 'compensatory.' 

• Should ease-of-computation*of the index be considered. As will become apparent in 
this report,^ome indexes can be/computed very simply, while others^?) be utilized 
ordy if one ViSs access to a computer. 

These indexes will be discussed in sequence, beginning with a relatively simple one and 
proceeding through progressivetynnore complex and sophisticated ones. 

t 

Targetingyrf CE services can be traced, in theory, from the legislation that makes the funds 
available, through the relevant 'regulations on how the funds are to,be allocated to states, 
districts, and then'$chool£ up to the schools' allocations of servic^fo .s^jdentl (Other 
^papers in this volume address the early steps *in this process.) For the purposes of this study,, 
only the last step in the allocation process, targeting within schools, will be considered. 
Because targeting at that level usually depends on academic needs, each targeting index 
will involve some measure of agreement between the selection of students for CE services 
(or their receipt of them) and a measure of the academic tteedinesS of the students. 

In the^ first section we consider different but related targeting indexes. These indexes 
represent the proportions of students selected for CE setvices who are^n need of thepi. 
Need is defined with several cutoff points on the nationally normed score distributions of 
reading and math achievement tests. The proportions can range from 0.0 (no student 
selected for the CE services at or below the achievement-score cutoff) to 1.00 (all selected 
students at or below the cutoff). These indexes are basedjbrunational definitions of need 



(the cutoffs) and notion local school definitions. They do not consider the neediness of 
students not selected for CE. 

, Indexes based on correlations between measures, of need for CE andyneasures of selection 
for CE (or receipt of instructional services) are investigated in the second Section in an effort 
to consider the needs of both CE and non-CE students. Indexes for schools are always more 
•restricted in range than those computed for grades, and indexes for reading are always 
higher than those for math., Although few of the values of the various indexes based on 
coefficients are negative "(indicative of gross mis-targeting), few of them are positive and 
sufficiently large enough for statistical significance at the 0.5 level (indicative of non«chance 
targeting). When need for CE is defined by teachers 4 judgments instead of test scores^ the 
values are much higher, 'but such indexes are confounded by a lack of independence 
between the judgmenrand the teacher's knowledge of e^ch student's CE status. On logical 
grounds, adjusted correction' coefficients ought to be more valid ^s targeting indexes; 
when computed they.also tend to be higher in value. ^ ^ ' 

When measures of the amounts of instructional services received are used instead of a, 
mere^dichfitomous indicator of CE participation, the values of the targeting indexes are 
much lower This is caused by the consideration of all schools, even those 'without tE 
(likely to be minimal, especially for reading, because almost all schools flad CE students for 
reading), and by the fact that amounts of services received by CE and rion-CE students 
overjap a great deal. 

An even greater refinement of targeting indexes may be found in the third section, where 
the dichotomous measures of student need for services (the use of cutoffs or teachers' 
judgments of need or no need) are replaced by a contfnous scale of percentile scores. The 
improvement in the size of indexes, -however, is, tittle more than one could expect merely 
statistically from coefficients based on a finer scale of measurement. The problem remains, 
however, that the use of a nationallyrfiormed' metric of n?ed simply cannot accurately 
reflect the degree of targeting within a school, when each school must target its services on * 
the basis of need relative toTf? student body. "* \ 

To takfe this relative need into account, the fourth section develops some additional 
indexes based on the achievement ranks of students within schools. Several related indexes 
are investigated that yield different values. A correlational approach relating relative need 
ranking to CE participation appears to offer the most sensible ^nd usable inde>^ of targeting. 
From these indexes we can conclude that about 75 percent of the schools target reading CE 
juen beyond chance levels, aftd abqjut 50 percent target math CE as adequately. When 
grades are considered, however, less than one-third of the first grades are well targeted for 
rea&Qg/>r matl). At the higher grades, about 75 percent targe^ reading CE well and about 
60 per<e»Wafget math well. 



INDEXES BASED ON THE PROPORTION O^CE PARTICIPANTS WHO ARE 'NEEDY' 

j 

The first approach to a targeting index 4e- be considered is based on the minimal, 
fundamental assumption of CE: namely, that all students being seJJ&ed are educationally in 



need of services. The mo£t bbvious index designed te reflect this is the proportion of CE 
students in a school who are educationally needy, where peed is defined by scores on an 
athievement test. In cases where all participating students meet the dritenon, the index 
value will be 1 0, when none of the students meet the criterion, the index value will be 0.0. 
The index ranges from zero to one, with intermediate values reflecting the degree to which 
- targeting among the CE students is appropriate ' * 

The fcTBS readi ng and math criterion scores were selected to reflect educational heed 
in the calculation of these proportions. Three different tfutoff ^co^s were selected, 
and anlyses performed in which students were classified as needy or not needy on the basis 
ofea£h.\ * <^ 

Below-Median Scores as the Definition of Need. TheJirst and highest of the cutoff 
f>oints ft the 50th percentile, the median. This score represents an extcemely high cutoff for, " 
the definition of need. We would expect to find agreement that children &ormg^t or 
above a median ba§ed on nationatnorms belong to the not needy' category, but it is not so - 
obvious that the cutting score is low enough— it is v not likely that we would find agreement 
that'all students scoring below the median are needV-' (Moreover, it will not likely ever be 

. economically feasible to provjde costly CE programs taone half of the student population.) 
Nonetheless, we, Start with the median to provide a point of reference. CE students for 
reading and math^were classified as needy jf they scored below the national median, and as 
not needy if they scored at or above it. The number of CE students at each school scoring 
below the national median was then divided by the total number of CE students ajt that 
school, to generate the index for the school. The same procedure was repeated for each . 
grade. Cumulative frequency distributions of the index for school¥^nd for grades are 
'presented in liable 7-1 for reading and for rqath. The indexes were based on data from each 
school in the SES first-year sample (for the schpol-level indexes) or e^ch grade (for the 
grade-lever i nndexes) in which v there were C£ students. Notice that tte;7ntervals of index \ 
values in Table 7-1 (and other tables) stb^at 0.96. By subtracting the cumulative percentage 

* at (hat value from 100.0,. the reader can quickly determine the percentage of schools (or 
grades) that exhibit perfect or near perfect targeting. * 

For reading, fewer than 3 percent of the schools had index values (first column in Table 7-1) 
* l£ss than or equal to .5d (indicating that at least half of the CE students were not needy. by 
our critieron), and almost 15 percent scored at .96 or* more (indicating alm % (& perfect 
targeting). For math, the corresponding cumulative percentages of schools were generally 
, higher, which indicate that targeting for math 1 CE is poorer (except that more schools seem, 
to have near perfect targeting for math). For both reading and math, the first grade shows 
higher percentages of schools in the lower range, 'especially near zejp, than any* other 
grade, probably reflecting low test validity at that grade. * 

The 40th and 35th Percentiles as Definitions of>Need. Additional cutoff points, lower 
than the median, were also examined with the hope that one of them jnight meet with 
wide^^greement as a meaningful or useful definition of educational need. A guideline can 
be obtained from another report (Kenoyer, "Teacher Judgment of Need for Compensatory 
Education," this volume), however, in which we find that th>e cutting score that minimized 
the disagreement between teacher classification and test classification varied somewhat 
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Table 7-1 



Cumulative Percentage Distributions by School and Grade: , 
Proportions of .Students in Reading CE and Math CE Who Score 
Below the 50th Percentile in Those Subjects 



v/ 









* 

Reading 








Index 
Value 


School 
(N«206f 


. Grade 1 * 
(ISU149) 


Grade 2 % 
(N-176) 


Grade 3 
(N* 180) 


Grade 4 
(N- 169) 


Grade 5 
(N-160) 


Grade 6 
(N = 146) 


0.(XX0 08 
0 09-0.16 
0 17-0.24 
025-0.32 


0.0 
0.0 
0.0 
■ fc 0.0 


6*6. 
* g 6.7 
8.7 
10/1 


1.1 
1 1 

'1.7 
2.8 


0.6 
06 
1.1 
1.1. 


t.8 
1.8 • 
1.8 

" t 1.8 


0.6 
-0.6 
0.6 
1 2 


' 0.7 
0.7 

0.7 \ 
0.7 


0.33-0 40 

0 41-0.48 

0 4&-0.56 

Q 57-0.64 
* > 


05 
1 5 
34^ 


/^8 ' 
/ 16.8 
( 28.2 
y 43.0 


10 2 , 
14 2 ' 


1.7 

*\^2 
4>- 
•8.9 


5.9 
6.5 
' 8 3 
11 8' 


1 9 

^ 1.9 
5.0 
7.5 


1 4 
1 4 
4.1 
4.8 


-0*5-0 72 

o 73,0.80 

0.31-0 88 
*0.89*0 96 


17 0^ 
28.6 
'52.4 • 
* 85.4 


77 2 \ 
88 6 1 
; ^91 3 * 


. 18 8 

27.3 ' 
41.5 

54.5-" ' 


128 
• 22.2 
37.2 
'49.4 


14.8 
22.5 
34.3. • 
48.5 


13.1 
19.4 
31 3 
, 45.0 


9.6 
17.8 
29.5 
- 42 5 


j - 


m 


o 


Math 




\ 




Index 
Value 


School * 
(N = 16ir 


Grade 1 
(N = 90) 


Grade 2 
(N= 100) 


Grade 3 
(N = 114) 


v Grade 4 
(N-113) ° 


Grade 5 
(N«112) 


Grade 6 
(N - 107) 


* 0 00-0 08 

0.09-0J6 
' n 1 7 n ">a 

0.25-0.32 ^ 


00 
0.0 

<r U U 

1 2 


4 4 ' 
4 4 

6 7 
»8.9 


3.0 
3.0 
4.0 
5*0 • 


2.6 
" 4.4 
4.4 

5.3' 


* 0.0. 

0.9 
1.8 . 
5.3 


' 0.0 1.9 

0.0 0.0 
*4 00 0.0 . 

"00* # 0.0 


0.33-0.40 
0.41-0 48 
0.49-0 56 
0.57-0 64 


3.1 
5.6 
9.9 
14.9 


T2.2 k 
V4.4 
30 0 
4^1 * 


8.0 
' * 9.0 
18.0 
23.0 


^7.0 
7.0 
13.8 
167 


7.1 
7. 1 
13.3 
21.2 


.0.0 
2.7 
5.4 
10.7 


2.8 
4.7 

5.6 * 


.0,65-0.72 
0.73-0 80 
081-0 88 
0,89-0 96 


25.5 
38.5 
62'1 
807 


53.3 
61 1 
70 0 
80.0 

* 


30.0 
47.0 
57.0 
61 0 


25.4 
30.7 
43.0 
58.8 


28 3 ' 
38.9 
k 53.1 
63.7 


16 J 

30.4 
42.9 
- 52.7 


19.6. 
30.8 
42.1 
55.1 



between reading and math" and^ver grades, but was bracketed fairly well bythe 35th and 
40th percentile. The 40th and 35th percentiles were therefore used as definitions of 
educational need-in the calculation of targeting indexes. Tables 7-2 'and 7-3 contain 'the 
v distributions based cji dichotomizations^at the 40th percentile and the 35th .percentile 
respectively. Inde\ values based on these cutoffs are, of course, lower than those based'on 
bhe 50th percentile, because the criterion is more stringent (a, greater proportion of CE 
•students must come from the lower achievement levels). Using the 35th percenffle, for 
example, only 1 percent of the schools exhibit perfect or near perfect. (.97 or'hig'hefl 
targeting for reading, and about 9 percent do sp for math (whereas for the median as cutoff, 
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Table 7-S 



• Cumulative Percentage Distributions %y ScHool and Grade: 
Proportions df Students in Reading CE and MathXE Who Score x 
Below th^ 40th Percentile in Those Subjects v * 



' Reading 



Index ' 
Value 



^hool 
(N-206) 



Grade 1 
(N*149) 



* -Grade 2 
- (N - V76) 



, Grade 3 * 


Grade 4 


Grade 5." 


Grade 6 


(N*180) 


(N-169) 


(N«160) 


(N* 146) . 


^ i.r 


3.6 


1.2 " * ' 


-2.1 „ 


' • '1.7 


4.1 


1.2* 


' 2.1 


' 2.2 


6 4,7 


* 1.2 \> 


2.7 - - 


« 3.9_ 


.6.5 ' 


1.9 


, 2.7* * 


, '/7.8' x 'e 


' i *y* 


3.7 - 


5.5 


8.9 


107 


5 6 


6.8 . 


' 16 1 ' 


1A2 


• k 137 \\. 


J„2.3 1 * 


21 7 • 


, 20/1 


v 19.4 


' 13.1 


31 7 * 


■> 28.4 . " 


• 28.1 


v 26.0; ,> 


43.9 


39.6 


48.(t • 


f 40.4 


55.0 


'56 8 


53.7 c v 


58.9* 


689 


4 66.3 * 







0.00-0.08 
0.09-0.16 
0.17-0.24 
0.2^-0 32 

0 33 0 40 
0.41-0 48 
0 49-0 56 
0 57-0 64 

'0.6&72 
0 73-0.80 
0 8Jr0 88 
• 0 89*0.96 
. L_i 



0.0 
0.0 
'0 0 
1 5 

24 
63 
126 
24 3 

*7 4 

56 8' 
81 1 
95.1 



7.4 
8.1 

to. I 

13.4 

21 5* 
28 9 
44 3 
63M 

79.2 
89 3 
%0. 
96.6 



2 3 
3.4 
5.1 

6 3' 

i 

102 
15.-3 
24 4 , 
'^8 

.38.6 
56 8 
•71 6' 
'80 1 



Math 

Index School Grade; Grade 2 firaoV3 

Value ' (N«161) : (N = 90). (N=100)' (Nf » 114) 



Crade 4 . Grade 5 

(N= : 1J3), *. iN^f12)' 



Grade 6s$ 

^(N«107) 



r p oo-o oa 


0.0 * 


6.7 % 


50 


Q09-0 16 


00 * 


78 , 


K 6.0 


0 1>-0.24 


06 


11,1. 


. 8.0 


J' 0 25-0 32 


5 0. 


15.6 


J2.0 


0 33-0. 40 


75 


20 0 I 


. 15 0^ 


■ 0.41^0 48 


13.0 


23.3 [ 


18.0 


0 49-0 56 


180 


43.3 


. 31.0 


0 57-0.64 


* 26 1 


54.4 

< ' * 


39 0 


0.65-0.72 


" - > \ 
49.7 


667 


50.0 


.0 73 080 


,68 3 m 


74.4 


66.0 


0.$ 1-0.88 


'82.0~ 


. 78 9 


75.0 


0 89-Q.96 


90 1 


84.4 


. 76.0 











"5 3^ * 


-0.9 


# 


27 


2.8'. 


7.0 


1.8 




16^. 


" 2.8 ' 


79 


4.4 / 




4.5 


' 2.8 


* 10.5 






< 4.5 


* 47 




* 




/ 




13.2 / i 


11.5, 




S.4 


* / 9.3 


16.7 


12.4 




7-1 


. • VI. 2 


"18.4 " „ 


25.7 




.17,9 . 


18.7 


27.2 


33.6 


t 


22.3 " 


27.1 


17.7- 


49.6 


*>- 


-35.7/ 


38.18 


52 6 


58.4 




50.<V* 


53 3 


62.3 


'* 74.3 




,65.2* 


:-"'64.5 


70.2 


7J.0 




67-9 L ' 


69.2 













the percentages of schools 'doing that well are 15 and 1^f. The achievementTtest definition 
of educational need one adopts clearly leads to different conclusions regarding how welf * 
schools are targeting CE. ' . ' ^ ^ / * ' 

^^erious shortcoming of the indexes in this section is that they depend only on the studerte. 
who participate in CE, and therefore a school having only low-achieving students can't help * 
but have fJerfect targeting (an index of 1.0). Therefore, the indexes in the next,section were 
developed using both CE and nOn-CE students. _ * * 
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Table 7-3 



Cumulative Percentage Distributions by School and Grade: ^ 
Proportions of Students in^Reading CE and \tath CE Who Score 
t Below the 3Sth Pekehtile inlhose Subjects 



kidex 



School 



iRe^ditig^ 



Grade 1 



Grade -2 



Value 


(N - 2%) 


'(N = 149) 


(N-176) 


,,.0.00-0 08 
0 09-0.16 
* 0*17-0>2£ 
.0 25-0.32 


■ °° 

ao ' 

0.5 
*% * 2 4 


-11.4 
' 134 
17 4 
-*26J 


,34 
- * . 4.0 
. 9.1 * 
11.9 


. <n3-o:4a 

* 0 41-0'48 
0 49-0 56 
0 57-0,64 


7 3 - 
136 - 

« / 28 3 " 


/O 6 
}8 4 ' 
72 5 
. 805 


1^ 9 • 

, ^24.4 
36.9 
*40<3 a , 


0 73-0 80 
0.81-0 88 
0*89-0 96 


■ji ■ * « . 
' 55 8 

93 2; 
/ 99 0 • 


89 3 * 
98 0 
"980 
• $8 0 


v • 

•> 

70 5, < 
80 7 - 
- 88.§ t 



'Grade 3 
* (N*180) 



Grade 4 4 



. £rade 5 
(N s? 160) 



Grade 6 
(N- 146) 



• -1.7 
2.2 

• : >.3 

' 50 
, i 

ll 1 

;'22.8 , 
M| 28.9 

;M2 8 
54 4 
70 6 
80.6 



4.1 

5.9 
. '6.5 
8 9 

130 

, 23 7 1 
' 2$h 

40.8 
58.0 
74 6 
79.$ 



< Math ' * ' 

lnde\ ' /School . Grade! Grade 2 * Grade 3 Grade 4 
Value* * * [N ? 161) (N = 90) (N=100) r (N=114) (N=113) 



K9 


- ' -2.7 


2.5 / 


3.4 


2 5 


' 4:8 


3 !■ ' 


48 






5 6 


I 9 


* 8.1 s 




■1?5 ' 


17 1 


*>5 6 j 


. , 30.8 


39.4 


1 

40.4 


53 1 


55.5 


64 4 


69 9 . . 


72 5 


74 7 


r 



GjtjadeS 
(N= 112) 



Grade 6 
(&?107X 



> 



0 00-0,08 
.0 09416 
0 17-0.24 
Ol25-0»32. 

0.33-040 
0.41-0 48 
Q 49-0 56 
0.57-0.64 

0 65-0 72 
0.73-0 SO 
•0.81-0.88 
' 0.89-0 % 



0<0. 


8.9 


70 


, 5.3 


4.4 


2.7 


06 


' 14 4 


10.0 


7.0 " 


• ' 6 2 


• „ 4,5 


5.0 


• 21 1, 


. 16 0 


9.6 


10 6 


4.5 


. 7 5 




190 


~ 10.5 ^ 


10.6* 


4.5 


9 9 


278 


* 23 0 


14 9 r - • 


18.6 


7.1 


17 4; 


.'40 0 


27.0 


19.3) 


21.2 


7.1 


25-5 


' ' 55 6 


39.0 . 


21.9 


32.7 


196 


41.6 


62,2 , 


48 0 


32. * % 


-3*9 8-' 


27 7 


60 2 


72 2 


* 

6CU> 


44.7 


60.2 


39.3 


75 2 


* 78.9 




*57 0 - 


71.7 


55 4 


85 1 


85.6 


'81 ft 


66 7, * 


8^:3 


68.8 


90 7 • 


- 8D/6 
> " 


; *8i o 


70.2 


83.2 


^ "72.3 


— , ' + 






t 







3.7 
3.7 
A 6.5 
15 

12.1 
17.8 
27.1 
29.9 < 

47.7 
64.5 " 
74.8 
78.5 



WITH SERVICES 



.One shortcoming -of" the indexes deyelope'd in .the previous section is that they. -are 
unaffected by the incidence of educational need among the sjudents not participating in 
CE Correlation* that involve neediness and* CE status, on the other hand, measure agree- 
ment between the two variables, with non-participants included, as well as participants. In 
this section, dicjiotomo'us indicators "Of nee"d are correlated with CE status first, then with 
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measures of amount of instructional sea ices received Where the dichotomous need 
indicators are correlated with CE status, 'itseffVdichqjomous variable, the resulting coeffi-, 
. cient is a 73/7/ coefficient Where th^ 1 need 'indicators are correlated with the quan-* 
* c titative variables indicating amount of service, there obtains a pomt-bisenal coefficient. 

« 

" Correlations with Indicators of Need ' 

r ? ■ 

The refinement^ introduced with these additional indexes comes from consideration of, 
students not selected for GE— students ignored in. the calculation of the previous indexes. 
' First, we consider a student's need for CE (or not) in terms of test^ scores and vvh^tbef or 
not the student participates. .Conceptualized in this way, the index will show- a perfect 
value when. all the CE students are needy (according to test score) and all non-CE students 
are not needy. • * 

Unadjusted Phi Coefficient. The kind of targeting index to be considered first is the 
- phi coefficient, the p/oduct-moment correlation between Tne dichotomtfbs variable 
t .denoting each student's selection for reading CE (or math CE) and the dichotomous 
'variable derived by cutting»the student's CTBS 1 reading (or math)" score at one of the three 
levels usedjn the^preceding section, i.e., th^ 50th, 40th, or*35th percentjle. There is 
fio, a prion basis '/or knowing with certainty which of the three dichotomtzatfbns Will 
yield the largest values for this index, since both CE and non-CE students are incfbded in 
the calculation. . 

The theoretical range of the phi coefficient is from -1.0 to + 1 .O.Therefore, while the 
■ ta'bles 4n this section are similar to earlier tables, the range of the index values is Afferent, 
v^Tabte* 7-4, contains, the cumulative distribution of the index (phi ' coefficients) 'for 
schools and. for each grade, for reading and math, where educational need is defined by 
djchotorrwzing the CTBS scale 'scores at the 50th percentile, basedon national norms. 

Looking first at threading distributions, we find that nearly all values are'positive, that only 
1.5 percent pf the" indexes at the school' level are negative, and that none are large. 

-Within this range, the distribution is fairly symmetricarand bell-shaped, indicating a wide 
range of ^targeting accuracy. The distributions by grade have a wider range, with* 
greater frequences of ektreme\alues iirbotJi directions. The difference in^anges is prdb- 
ably the result of the aggregation process that leads to greater reliability at the school level. 

, Grade 1" has more tow index values, w*ithck>se*to-20 percent of them less than zero. This js 
attributable, at least in part, to low test validity at thi? level, but is also probably influenced * 
by lower, accuracy of CE assignment fcom a relative unfamiliarity wjth student capabilities. 

The distributions are a bit different for math, with the lower end of the range somewhat 
more negative at the school level." The same relative disadvantage in targeting ahthe first- 
grade level is apparent, and the mdfex values by grade again tend to have -greater scatter 
than the school-level values. This finding suggests' Wat targeting is generally" better at the 
school level because the student selectioruis less restricted. The school-level^ indexes, of 
course, are lowered when schools limit CE services to,selected„grades while there^are many 
needy students in the excluded ^fade's. . ' . * • * * 
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Table 7-4 

Cumulative Percentage Distributions by School and Grade: 
Phi Coefficients Between CE Participation and Achievement 
Scores' Dichotomized at the 50th Percentile 



Reading 

Index School Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 

Value (N-205) , (N ■ 143) (NU168) (N-172) (N-143) (H- 155) (N-144) 



- 62 - - 53 


00 


0.0 


0.0 


-.52 - - .43 


00 


0.0 


0.0 


-.42 - -.33 


0.0 


1.4 


0.0 


-.32 -'-.23 

* 


00 


5.6 


0.0 










- 22 . 13 


00 


8.4 


0.6 


- 12 - - 03 


1 5 


19.6 


3.6 


-* 02*- 01 


. . 2.0 


4 25. 2 


4.2 


03- 12 . 


7.'3 


• 51 7 


11.9 


.13- 22^ 


26 8 


72-0 


22.6 


23.32 


50.2 


' 888 


'33.9 


.33-.42 


78 5 


94.4 


48.8 


.43-.S2 


95.1 


v 97.9 


66 1 


.53-* 62 


99.0 


98.6 


77.4 


.63- 72 


995 


98.6 


89.9 


..73- 82 


1000 


993 


95.8 


83^92 




99.3 


98.8 



\ 



0.0 0.0 * 0.0 0.7 

0.0 0.0 0.0 0.7 

o:o - ao o.o * ~o.7 

0.0 0.0 0.0 0.7. 

1.2 ] 2 0.0 ' 1.4 

2.3 18 1.3 • 2.1 
2.9 3.1 2*6 2.8 
.5.2 9.2 , 7.1 - . ,4.9 

19 2 - b 22.1 1 % - 18.1 ' 23.6* 

j> . _ 

33.7 37.4 33.5 50.7 

47.7 57 7 54,8 66.7, 

68.6 75 5 > 74.2 8f.9 
87.2 92.6 87.7 "94.4 
92.4 * 98.1 95.5 97.2 

97.7 ' 99.4 >• 98. T 97.9 
98.2 99.4 v , 98.7 98.6 



Math 



Index 


School 


Grade 1 


Grade 2 t 


Grade 3 


Grade 4 


Grade 5 


Grade 6 


Value 


• (N-161) 




(N-96) 
—> — 


(N- 109) 


(NU100) 


(N « 107) 


(N-104) 


-.6^ • - 53 


/O.O 


1.2 


00 


0.9 


0.0 


0.0 ' 


o.cT 


-.52 - -.43 


• 0.0 


1.2 


0.0 


. 0.9 


0.0 


0.0 


0.0 


. - 42 - -.33 


00 


4.7 


0.0 


3.7 


0.0 * 


00 


0.0 


-.32 - - 23 


00 


4 7 


1 0 


'4.6 


09 


» 00 * 


0.0 


^22 - - 13 


1.2 


8.1 


3.1* 


6:4 


1.9 


0.0 * 


1.9 


-.12 - -^.03 


\* 6.8 


19.8 


< 9.4 


9.2 


5.6 


0.0 


3.8 


-.02-.02* 


9.9 


29/1 


11.2 


11 9 


7.4 


0.9 


- 4.8 


.03-. 12 


29.2* 


40.7 


• 24.0 


261 


19.4 


10.3 


> * .21.2 


.13-.22 


54.7 


67.4 


' 49.0 


v 41.3 




32.7^ . 


47.1 


* .23-.,32 


* 

84 5 


v . 83.7 


70.8 „ 


62.4 


67.6 


62.6 


75.0 


J3-.42 


* 93.2 


89.5 


90.6 


78.0 


82.4 


80.4 


84.6 


.43^.52 


96.9 


95.3 


95.8 


87.2 


89.8 


' 85.0 


93.3 


52- 62 


98.8 


97.7 


96.9 


94.5 


97.2 


93.5 


97.1 


63- 72 


99.4^ 


100.0 


99.0 


96.3 


98,1 


98.1 


100.0 


73- 82 


100.0 




99.0 


96.3 


100.0 


99.1 , 




.83-.92 






100.0 


97.2 




99.1 





f 

Note. The indexes m the middle set of rows of each subtable (- 22 to + 22) have values'that are* (approxi- 
mately) not significantly different from zero—or chance targeting. " s 



ERJC ' 93 121 



f 

J 



Table' 7-5 contains the same distributional information, based on the 40th percentile 
dichotomization, and fable 7-6 is based on the 35th percentile cutoff. The three* cutoff 
scores, when compared, do not lead to markedjy different distributions.-Summaries of the 
school-level distributions are shown in Table 1-7 for the purpose of comparison of ihe 
effects of the three "dichotomizations. For bQth reading and math, the 35th and 40th per- 
centile cutoffs yield higher index values than when the cut is at the median (50th 
percent! leY Targeting of reading CE is clearly better than targeting of math CE, but there are 

many instances of ineffective or poor targeting in both cases. 

/ « 

u 

Taking the median value for reading among the three cutoffs li.e.,,the median for the 
phi based on the^40th p^rcent^le cut), we find that ab6ut 10 percent) of the variancfe of 
the dichotomized* achievement sc^re and CE assignment is shans^d. The best math mean, 
based on, the 35th percenyl^cut, indicates that about 4 percent of the variance is sharecfby 
the two variables The implication is that this kind of index cannot account for the grade- 
by-grade and schoSI-by-school variation in how students are selected for CE' Unless a 
strict and universal selection criterion were mandated, this kind of index is likely to indicate 
poor targeting. 

- r . 

Unadjusted Phi .CueriiLient with Teacher Judgment ot Need, tor CE. In addition to the 
% indexes abqv^, all based on dichotomiAitions of the CTB5 scores, a different index can be 
"created by replacing tbe Q'BS^dichotomies wifh teacher judgment of each student's need 
for CE A targeting index was derived for reading by correlating teacher judgment of need 
for reading CE (needs it or does not need it] with whether or not the student participates in 
reading'CE, and similarly for math In attempting to evaluate these indexes in relation to the 
CTBS indexes, it is important to be aware that in many cases the teachers probably knew 
the CE status qf each student, thus contaminating their responses. Therefore, agreement 
beiween teacher judgmef^ and CE status may be. spuriously inflated, ancj these indexes 
mu>t be interpreted carefully. 

Table 7-8 contains the distributions of these indexes for reading and math f5r schools and 
for grades within schools For both readings and math, the correlations tend to be more 
positive, for reading, there are more correlations greater than 80 than there are negative 
correlations m any grade. For math, the positive tendency is not as strong, but* it too is 
clearly present. The teacher-judgment criterion of need for CE serves as a somewhat better 
correlate of CE K status that the CTBS dichotomies, but it is impossible, given the circum- 
stances' of d$ta collection, to rule out spurious inflation of the coefficients. 

* Adjusted Phi Coerticienh A problem with usjng the phi coeffiqienr as a targeting 
index is that the theoretical range from* -1.0 to +1.0 can be achieved only if the two 

> N diChotomous variable 6eing correlated have equal marginal probabilities. For the present 
analyses, . the departure from that condition can be expected to be severe, the number of 
students who can be selected for CE is influenced by a schooIVfunding level, among other 
things, and the number of needy students is not under the control of the school (at least * 
initially). Since our focus is upon the effectiveness of school targeting .procedures, and 
^ecause it seems appropriate \o let the extraneous influences enter into the value of the 
targeting index, we should examine an adjusted coefficient. 
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Table 7-5 



Cumulative Percentage Distributions by School and Grade: 
Phi Coefficients Between CE Participation and Achievement 
Scores Dichotomized at the 40th Percentile 









Reading 






0 


Index 
Value 


School 


Grade 1 

(IN = 14J) 


Grade 2 
(N-167) 


Grade 3 
(N= 172) 


Grade 4 
(N=163) 


Grade 5 
(N-154) 


Grade 6 
(N = 144) 


-.62- - .53 
- .52 • -43 
-.42-,-. 33 
-.32- - .23 


0.0 
00 
0.0 
0.0 


* 

0.7 
07 
2.1 
4.2. 


0.0 
0.0 
0.0 
0.0 


0.0 
0.0 
0.0 
0.0 


0.0 
0.0 
0.0 
0.0 


0.0 
O'.O 
• 0.0 
0.0 


0.0 
0.0 
0.0 
0.0 


-•22 -13- 

- \Z - - 03 

- ,02- 02 
03- 12 
13-22 

A 


00 

■ a 

6 3 
20.5 


9.1 
21.7 
31 5 
53.8 
727 


0.6 
36 
54 
14 4 
20.4 


0.6 
1.7 
29 

A A 

14.5 


0.6 
j ^3.1 
J 49 
8 0- 
19.6 


1.3 
1.9 

3.9 1 
5.8 
14.3 


0.7 
2.1 
4.2 
7.6 
24.3 


23-. 32 
33-.42 
.43- 52 
.S4-.62 
.63-72 

' .83- 92 


468 
76 6 
' 94.6 
,99 5 
100.0 


88 1 
93.7 
' 97.9 
97,9 
99.3 
99.3 
99.3 


29.9 
48.5 ' 

A 7 1 
O/. 1 

82.0 
92.8 " 
98.8 
100.0" 


25.0 
45.3 
63.4 
83.7 
91.3 
96.5 
98.8 


34.4 
.51.5 
- 72.4 
85.3 
95.1 
- 98.1 
98.8 


^28.6 . 
43.5 
67 5 
83.1 
92.9 
96.8 
98.1 


41.7 
61.1 
79.9 
90.3 
97.2 
100.0 


* 


• 




Math 










Index 
Value 


School " . 

(hi — 


Grad^ 1 
(IN =»o/J 


Grade 2 
(N«98) * 


Grade 3 
(N=108) 


Grade 4 
(N= 108) 


Grade 5 
(N » 109) 


Grade 6 
(N = 104) 


- ,6'2 - .53 
-.52 - - .43 

- 42 - - .33 
-.32 - - 23 


, 0.0 
0.0 
0.0 


0.0 
1.2 
3.5 
5.8 


'o.o" 
0.0 
0.0 
0.0 


0.0 
0.0 
0.9 
37 


0.0 

p.o 

.0.0 
0.0 


*0.0 
0.0 
0.0 
0.9 


0.0 
0.0 
0.0 

o.o 


-.22- -.13 

- 12 - - m 

- .02-.02 
.03- 12 

13-.22 , 

c 


06 > 

A 1. 

75 
27.3 
55 3. 


8 1 

15 3 
30.2 
41.9' 
67.4 


2.1 

7.3 

29.2% 
47.9 


37 
■ ' 8.3 
12.0 
18.5 
36.1 


0.9 
5.6 • 
8.3 

23.1 

47.2 


0.9 
3.7 
' 4.7 
13 1 
29.9 


1.9 
6.7 
10.5 
23.8 
44.8 


23- 32 
33-.42 
M3-.52 
.53- 62 
.63- 72 
73-,82 
.83-.92 


*82.6 

93.8 

98.1 , 

99.8 
1000 


83.7 
93.0 
95.3 
97.7 
100 0 


65.6 
802 
93.8 
95.8 
99.0 
99.0 
100.0 


56.5 
77-8 
83.3 ' 
90.7 
,96 3 
97.2 
97.2 


67.6-* 
81.5 

89.8 . 
96.3 
99.1 
100.0 


. 53.3 
75.7 

86.9 ' • 
92.5 • 
99.1 
100.0 


71.4 
82.9 
95.2 
97.1 
99.0 
.100.0 



Mote- The indexes in the middle set of rows of each subtable (-.22 to +.22) have values that are (approxi- ' 
mately) not significantly different from zero— or chance targeting. 
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Tabled 

Cumulative Percentage Distributions by School and Grade: 
Phi Coefficients Between CE Participation and Achievement 

Scores. Dichotomized at the 35th Percentile $ 



- • 






Reading 








Index 


School 


Grade 1 


Grade 2 „ 


Grade 3 


Grade 4 


Grade 5 


Grade 6 


Value 


(N - 205) 


(N - 143) 


(N - 164) 


(N- 171) 


(N - 163) 


(N- 153) 


(N- 144) 


'-.62 - -.53 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


- .52 - - .43 


0.0 


0.0 * 


.0.0 


0.0 


0.0 


0.0 


0.0 


- .42 - - .33 


0.0 


1.4 


0.0 


0.0 


0.0 


0.0 


0.0 


-.32 - -.23 


o.tf 


4.2 


0.0 , 


0.0 


00 


0.0 


0.0 


-.22 - -*13 


0.0 


11.2 


1.2 


0.6 


0.6 


0.0 


1.4 


-.12- -.03 


1.0 


24.5 


3.6 


2.9 


3.1 


1.3 


3.5 


* - 02-.O2 


1.0 


33.6 


c c 
D.J 


A 1 


4.9 


2.6 


5.6 


.Q3-.12 


7-3 


58.0 


12.7 


i 7 -° 


8.6 


5.2 


10.4 


.13-^2 


23.9 


76.2 


^,23.6 


/1 8.1 


20.4 


12.4 


22.9 


. .2*. 32 


48.8 


90.9 


AJ1.5 


• 

30.4 


37 0 


26.1 


43.8 ° 


.33*. 42 


79.5 


95.1 


✓%2.1 


46.2 


C A 1 

54.3 


46.4 


61 .8 


A") CI 

.43-. 52 


. CIA C 

94. b 


* no c 
99. D 


72.7 


63.7 


71 C 
/l .6 


D/.J 


TC A 
/D.4 


CI C 
.jJ-.OZ 


QQ C 




84.2 


82.5 


Al 1 


at n 


Q1 7 


' .63-. 72 


99.5 


99.3 


Q7 Q 


Q1 1 


95.1 


93.5 


96.5 


\73-.82 


100.6 


99.3' 


QA A 

yo.o - 




96.9 


96.7* 


> 98.6 


.83-.92 


V 


100.0 


IUU.U 


Q7 7 


' 98.2 


97.4 " 


* 98.6 








Math 










Index 


.School 


Grade 1 


. Grade 2 


Grade 3 


Grade 4 


Graded 


Grade 6 


Valde 


(N ■ 161) 


(N-87) 


(N-98) 


(N - 108) 


(N - 108) 


(N - 109) 


(N- 105) 


-.62 - -.53 


0.0 


0.0 


0.0 


0.0 


0.0 


* 0.0 


0.0 


-.52- -.43 


0.0 


0.0 ' 


0.0 


0.0 


0.0 


0.0 


0.0 


M2- ,.33 


0.0 


1.1 


0.0 




0.0 


0.0 


0.0 


-.32 - -.23 


0.0 " 


4.5 


0.0 


2.8 


0.0 


0.9 


0.0 


-.22 - -.13 


1.2 


10.1 


2.0 


3.7 


2.8* 


0.9 


1.9 


-.12 - -.03 


5.6 


18.0 


11.2 


9.3 


9.3 


2.8 


4.8 


-.02-.02> 


8.1 . 


' 31.5 


15.3 


13.0 


10.2 


• 6.4 


10.5 


.03-. 12 


26 A 


48.3. 


29.6 


23.1 


25.0 


12.8 


20.0 


.13-.2r 


' 53.4 


66.3* 


49.0 


35.2 * 


41.7 


33.0 


51.4 


.23-/32 


.79.5 * 


83.1 


63.3 ' 


52.8 


68.5 


51.4 


68.6 


J3-.42 


93.2 


92.9 


80.6 


73.1 


82.4 


74.3 


81.9 


.43-.52 


98.1 


95.5 


90.8 


84.3 


92.6 


88.1 


91.4 


.53-.62 


98.8 


96.6 


95.9 


91.7 


99.1 


92.7* 


98.1 


.63-/72 . 


100.0 


98.9 


100.0 


96.3 


99.1 


98.2 


•98.1 


».73-.82 




98.9 




97.2 


100.0 


100.0* 


* 100.0 


.83-.92 




98.9 




97.2 









' Note: Tne indexes in the middle set of rows of each subtable (-.22 to +.22)*ftave values that are (approxi- 
mately) not significantly different from zero— or chance targeting. 
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For each phi coefficient it is possible to determine the. maximum value that it could 
take given the existing marginal frequencies. When the marginals are equal for the two 
•variables, the maximum possible, value for phi is 1.0. In general, especially ^vvhen the 
. marginals aren't equal, a theoretical maximum can be computed. Guilford (1965, p. 336 ff.) 
describes this ■> maximum as ph/ max , and points out- that dividing phi by 
P^'max serve? in part to correct the attentuated coefficient. The resulting coefficient is 
no longer a true correlation/ and its distributional characteristics are not all desirable, but 
desplte-theoretical shortcomings it is of interest in the case at hand to impose this 'correc- 
tion.' We can then, determine whether an.adjustment of this kind makes any important' 
difference, and so<draw some rough inferences about the impact of existing funding con- 
straints (that would cause unequal marginals, in part) on targeting. 



Table 7-7 

Cumulative Percentage Distributions by School: Phi Coefficients 
Between CE Participation and Achievement, "With Achievement Gfichotomized 
at Three Different Percentile Points for Reading and Math 



Index 
. * Value* 


> 




Reading 


£ 




Main 




50th 
Percentile 


40th 
Percentile 


35th 
Percentile 


50th 
Percentile v 


40th 
Percentile 


35tfc 
Percentile 


-.20 




0.0 


0.0 


0.0 e 


D.6 


0.0 


0,0 


- 15 




80 


00 


0.0 


"1.2 r 


0.6 


* 1.2 


- 10 




0.0 


1 0 


• 0.5 


o 3.1 


1.9 


1.9 


-.05 




1 5 


1 $ 


1.0 


6.8 > 


4.3 


• 5.6 


00 




• 2.0 


1.5 


1.0 " p 


9.9 


7.6 


8.1 


05 




3.9> 


r 29 


3.4 


16 1 


15.5 


13.7 


10 ' 




7 3 


6.3 


7.3 


29.3 


27.3 


26.1 


.15 




156 V 


13.7 


13.2 


43.5. " 


41.0 


42.2- 


.20 




, 26.? 


20.5 


23 9 


• 54 7 


55.3 


53.4 






s : - 












.25 




37.1 


33.2 


35.6 


^f-5 * 


'73.9 


68.3 


30 




50 2 


46.8 


- 48.8 




. 82.6 - 


79.5 


35' 




62.0 « 


, 60.0 


63.9 


90.1 


90.7 


89.4 


40* 




->8.5' 

* "% 


76.6 


79.5 


93.2 


93 8 - 


93.2 


/ .45 






86.3 


- 86.3 


95.7 


. 9*5.7 


, 96.3 


50' 




95.1 


94 6 


94.6 


96.9 


, 98.1 


98.1 


55 


98.0 


.98 0 


97.1 


98 1 


98.1 


. 98.8 


.60 




99.0' 


99 5 * 


98 5 


98/8 


99.8 


98.8 


.65 




99.5 


^9.5 


99.5 


98.8 


99.4 


99 4 


.70 




99.5 


100.0 


99.5 


994. v f 


100.0 


' 100.0 


7 5 




99.5 




100.0 


100 0 ' 


| .80 




100,0 













•Index values reported as midpoints of intervals of .05. 
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. Table 7-8 



Cumulative Percentage Distributions by School <md Grade: 
Phi Coefficients Between CE Participation and Teacher Judgment 
of Neejl for CE, for Reading and Math 









Reading 








Index 


School 


Grade 1 


Grade 2 


* Grade 3 


Grade 4 


Grade 5 


Grade 6 


Value 


(N » 205) 


(N-138K 


(N » 169) 


' (N * 1 72) 


(N » 163) 


(N»153) 


(N»143) 


-.42 - -.33 


0.0 


0.0 


0.0 


0.0 


ao 


0.0 


*r 

0.0 


-.32- -.23 


0.0 


1.4 


0.0 


~ 0.0 


0.0 


0.0 


o.o e 


-.22 - -.13 


0.0 


5.8 


0.0 


0.6 


0.6 


txo 


0.0 


-.12 • -.03 


2.0 


10,9 


1.8 


2.3 


2.5 


1.3 


1 2.V 


-.02-.02 


2.4 


13.0 


3.0 


2.9^ 


3 1 


2.0 


5.6 


.03-. 12 


4.9 


15.9 


4.7 


4.1 


3.7 


2.6 


8.4 


.13-.22 


, 8.3 


22.5 


8.9 


8.1 


7.4 


* 5.9 


11.9 




1 D. 1 


Zo.S 


16.0 


14.5 


1 4.1 


v 9,8 


23. 1 


.33- 42 


30.2 


38.4 


20 7 


23.3 


28.8 


22:9 


34.3 


.43-. 5 2 


44.9 


50.0 


32.0 


37.2 


46.0 


32.7 


- 46.9, 


.53-.62 


65.9 


67.4 


46.2 


47.1 


vc 57.7 


43.1 


62.2 


.63-. 72 


84.4 


76.1 


60.4 


62.2 


70.6 


66.0 


74.8 • 


.73-.82 


96.1 N 


87.0 


81.1 


9 80.8 


' 84.0* \ 


84.3 


86.7 


% .83-.92 


100.0 


94.9 


95.3 


92.4 


91 4 ' 


95.4 


96.5 


i ' ' ' 






Math 




• 






Index 


School 


Grade 1 


Grade 2 


Grade 3 


Grade 4 * 


Grade 4 


Grade 6 


Value 


(N>161) 


(N=<85> 


(N s 95) 


(N« 107) 


AN = 108) 


(N * 1Q7) 


(N = 102) 


-.42- -.33 


0.0 


1.2 


0.0 


. 0.0 


0.0 * 


0.0 


0.0 


-.32 - -.23 


■ 0.0 


4.7 


0.0 


0.9 


0.9 


0.0 


0.0 


-.22 - - 13 


0.6 


8.2 


1 1 • 


■ *3.7 


2.8 


0.0 


1.0 


-.12 03 


3.7 


16.5 


6.3 


- 8.4 


6.5 


6.5 


3:9 \ 


- 02-.02 


5.6 


17.6 


11.6 


9.3 


8.3 


8.4 


4.9 \ 


.03-. 12 


9.9 


28.2 


147 


14.0 9 


^ 12.0 


12.1 


6.9 


.13.22 


21.7 


38.8 


'24.2 


19.6 


20.4 


19.6 


15.7 


.23-.S2 


39.1 


45.9 


33.7 


29.9 


31.5 


30.8 i 


32.4 


J3-.42 


58.4 


57.6 


48.4 * 


- 43.9 


46.3 


43.0 


44.1 


.43-.52 


73.9 


6?.4 


67.4 


63.6 


62.0 . 


54^ , 


53.9 


.53-.62 


84.5 


75.3 


76.8 


74.8 


74.1 


673 V 


74.5 


.63'-.72 


94.4 


8.7.1 


83.2 " 


81.3 


84.3* 


80.4 


89.2 


.73-.82 


98.1* 


95.3 


90.5 


' 92.5 


92.6 


86:9 


93.1 


.83-.92 


, 99.4 


97.6 


95.8 


* 95.3 


£7.2 


93.5 ' 


95.1 



75 — ; : ' " • 

The next group of tables are organized and formatted like those of the preceding sections. 
Table 7-9 contains the frequency distributionsVor schools and for each graiJe, based on the 
40th percentile criterion for reading and for math. Table 7-10 contains the, reading and 
math f[equency distributions based on the 35th percentile. (The correlations derived from 
the 50th percentile tehded to be lower in the preceding section, and this ordering would 
not likely be changed by the correction procedure. The 50th percentile correlations were 
therefore excluded frpm the adjusted-p/?/ analyses.) 
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I Table '7-9 



* 

Cumulative Percentage Distributions by School and Grade: 

Corrected Phi Coefficients Between CE Participation 
and Achievement Scores Dichotomized at the 40th Percentile 









Reading 




! 






/ Index 


\ School 


Grade 1 


. Grade 2 


Grade 3 


Grade 4 


Grade 5 


Grade, 6 


f Value 


(N~205) 




/VJ _ -*£. y\ 


(l\ = 1/1) 


(N = 162) 


(Nj|154) 


(N« 144) 


Less than - 63 0.0 


49 


0.0 


1.2 


0.6 


0.0 


0.0 


-.62 - - .53 


0.0 


6.3 


0.0 


1.2 


0.6 


• 0.6 ' 


0.0 


-.52 • - 43 


0.0 ' 


8.4 


0.6 


1.2 


06 


0.6 . 


0.0 


- 42 ■ - .33 


0.5 


9.1 


1.2 


1.2 ' 


0 6 


0.6 


0.7 ' 


-.32 - - .23 


0.5 


- 11.9 


« 1.8 


1.2 


0.6 


0.6 


0.7 • 


- .22 - - .13 


U.J 


1 A ft 




1 0 
1 .0 


1.2 


1 .3 


0.7 


-.12- - 03 


1 n 

I u 


'22.4 


4.2 


1.8 


3.1 


2.6 , 


2.1 


1 - .02- 02 


1 n 

1 u 


Jv 1 


D D 




4. J 


3 9 


4 2 


.03- 12 


1 n 

£ U 


38.5 


90 


4.7 


4.9 


39 


4.9 


. 1 3- 22 


• 44 ' 


50.3 . 


11.4 


5.3 


6.2 


5.2 


9.0 


.23- 32 


in 1 


• 63.6 


15.0 


. 7.6 


12.3 


. 8.4 


13.2 




11 A 
11 4 


70.6 


19.8 


12.3 


17.9 


12 3 


160 


> .43-.52 


19 7 


7y.7 


27.5 


17.0 


24.1 


19.5 


22.9 


.53. .62 * 




00 1 
00. 1 


38 9 


30.4 


38.9 


29 9 




.63,72 


73.2 


9J.0 


53.9 


46.8 


5K2 


44.8 


3 ^ 
. 4*9 


.73-82 


85 9 


94.4 


64.7 


59.6 


60.5 


58.4 




ft! Q7 


95 6 


95.1 


72S- . 


64.3 


64.2 


63.6 


63.9 




\ 




Math 










Index 


/ School 


Grade 1 


Grade 2 


Grade 3 


Grade 4 


Grade 5 


Grade 6 


Value 


(N= 161) 


(Ki — QC\ 


\VS - JO) 


(l\ = 1UO) 


/vi ino\ 
(iN = lUo) 


(N = 107) 


(N = 105) 


. less t(*an - .6 


I . 0.0 


3.J 


0.0 


2.8 


0.0 


0.0 


0.0 


-.62 - - .53 


\ a.o 


8.1 


0.0 


3.7—- 




0.0 


0.0 


- 52 -/.43 


4 yn.2 


9.3 


0.0 


3.7 


0.9 


'0.0 


1.9 


-.42 /-.33 


1.2 


14.0 . 


■ 1.0 


3.7 * 


1.9 " 


0.0 


1.9 


-.32/- -.23 j 


1 1.9 


174 


4.2 


4.6 


1.9 


0.0 * 


2.9 


- .22 • - p 


•1 9 


22 1 


6.3 


5.6 


2 8 


7 ft 


l ft 


-Rcj/03 


4.3 


25.6 


8.3 


' 8.3 


7.4 


4.7 


6.7 . 


- 02- 02 


6.8 


29.1 


1 13.5 . 


10.2 


83 . 


4.7 


8.6 


03-12 . 


l\.8 


32.6 


19.8 


130 


, 15.7 


5.6 


, 12.4 


.13-.22 


15.5 


45.3 


20.8 ' 


16.7 


19.4 


10.3 


20.0 


.23,-32 


26.1 


53.5 


32.3 ' 


21.3 


3*1.5 


14.0 


27.6 


T33-.42 . 


39 8 


61.6 ' 


42.7 • 


27.8 


39.8 


22.4 


33.3 


.43- 52 


54.7 


67.4 


51 0 


36.1 


52.8 


37.4 


44.8 ' 


.53 .62 


' 67.1 


72.1 


58.3 


49.1 


62.0 


43.9 


52.4 


.63-.72 


76.4 


76.7, 


65.6 


55 6 


* 68.5 


55.1 


64.8 


.73-.82 


84.5 


80.2 


70.8 


61.1 


75.0 ' 


61.7 


66.7 


.83- 92 


88 8 


81.4 


70.8 


64.8 


75 0 


. 64.? 


68.6 



Note- The indexes in the middte set of rows of each subtable (-.22 to + .22) have values that are (approxi- 
*^ m'ately) not significantly different from zero— or chance targeting. 



ERIC 



x Table 7-10 



Cumulative Percentage Distributions by School and Grade: 

Corrected Phi Coefficients Between CE Participation 
and Achievement Scores Dichotomized^ the 35th Percentile 









Dointno 










index 


Crk/wJ 

jcnotM 


oraoe l 


oraoe z 


viraoe o 


* Grade 4 


oraae j 


oraae o 


Value 


(N-205) 


(N- 143) 


(N - 165) 


(N-171) 


(N - 162) 


(N-153) 


(N-144) 


Less than -.63 


. 0.0 


2.1 


0.0 


1.2 


0.6 


v 0.0 


0.0 


-.62,- -.53 


0.0. 


2.8 


0.0 


1.2 


1.2 e 


0.0 


0.0 


-.52 - -.43 




- 4.9 


0.6 


1.2 


1.2 


4 0.0 


0.0 


-.42 - -.33 


0.0 


8.4 


0.6 


1.2 


1.2 


0.0 


0.7 


-.32 - -.23 


0.0 


12.6 


1.8 


1.2 


1.2 


0.0 


0.7 


-.22 - -.13 


0.5 


16.8 


3.0 


2.3 


1.2 


0.7 


1.4 


-.12 - -.03 


1.0 


29.4 


3.6 


2.9 


3.1 


1.3 


15 


-.02-.02 > 


T.O " 


32.9 


4.2 


4 1 


4.9 


2.0 


5.6 


.03-. 12 


1.0 


46.2 


6.7 


5.8 


* 6.8 


3.9 


7.6 


^^x 11 


7 1 


bz.z 


11 c 


7 Ci 


in r 


3.9 


in J 

1U.4 f 


.23-.32 


15.1 ' 


70.6 


14.5 


12.3 


17.3 


7.8 


13.2 


J3-.42 


27.8 


81.1 


24\8 


16.4 


23.5 


11.8 


27.1 


.43-.S2 


49.3 


85.3 * 


32.1 


. 28.1 


35.2 


24.8 


33.3 


.S3-.62 


70.7 


92.3 


46.1 


40.4 


50.6 


41.8 


44.4 


.tt»72 


82.0 


95.1 


57.6 


59fe 


63.0 


58.2 


57.6 


.73-. 82 


92.7 


95.8 


69.1 


69.0 


74.1 


67.3 


69.4* 


.83-.92 


98.5 


* 95.8 


80.0 


74.9 


76.5 


70.6 


71.5 








Mam 










+ IllUCA 






vjraoe x 


iiraoe j 


Orade 4 


vjraoe j 


vjraae o 


Value 


(N-161) 


(N-86) 


(N-%) 


(N-108) 


(N - 108)» 


(N-107) 


(N - 105) 


Less than - .63 


0.6 


5.8 


0.0 


"2.8 


1.9 . 


0.0 


0.0 


-.62 - -.53 


1.2 


5.8 


0.0 


3.7 


1.9 


0.0 


0.0 


-.52 * -.43 


1.2 


8.1 


0.0 


■ 3.7 


1.9 


0.0 


1.9 


-.42 ---.33 


1.2' 


9.3 


1.0 


3.7 . 


1.9 


• 0.0 


3.8 


-.32 - -.23 


1.9 




4.2 


5.6 


2.8 


0.0 


3.8 


-.22 - -.13 


3.7 


17.4 


7.3 


6.5 


~\.6 , 


0.9 


4.8 


-.12 - T.03 


5.0 


22.1 


10.4 


9.3 


10.2 


2.8 


5.7 


-.02-.02 . 


8.1 


25.6 


11.5 


11.1 * 


10.2 


4.7 


8.6 


.03-. 12 


10.6 


36.0 


20.8 


13.0 


15.7 


5.6 


* 14.3 


.13-.22 , 


J 8.0 


43.0 


24.0 


17.6 


18.5 


9.3 


19.0 


.23-.32 


33.5 


53.5 


31.3 


24.1 


35.2 


15.9 


32.4 


J3-.42 


42.9 


60.5 


44.8 


28.7 


• 42.6 


25.2 


38.1 


.43-.S2 


58.4 


67.4 


53.1 


41.7 


54.6 , 


34.6 


43.8 


.53-.62 


v 72.7 


73.3 


61.5 


51.9 


68.5 


46.7 


58.1 


. .63-. 72 


78.3 


79.1 


69.8 


56.5 ' 


74.1 


59.8 


70.5 


.73-.82 


85.7 * , 


100.0 


75.0 


62.0 


80.6 


68.2 


76.2 


.83-.92 

4 


89.4 




75.0 


64.8 


80.6 


69.2 


78.1 



^Note: The indexes in the middle set of rows of each subtable (-.22 to +.22) have values that are (approxi- 
mately) not significantly different from zero— or chance targeting. 



Applied to our data, the adjustment produces a small percenfage of index values that are 
less than -1.0, -i.e., , less Jhan the theoretical minimum for phi coefficients. The inap- 

, propriateness of the correction for negative value* is 4 one of the major deficiencies in the 
correction procedure. Most of the valuer are positjye, however, and the correction clearly 
increases the phi values. For example* in the school-level, analysis for the 40th percen- 
tile cutoff, the median value for the adjusted phi is abdut .59, compared to about .32 
for the unadjusted phi. ^s a description of targeting, the adjusted phi is successful, 

Jn eliminating the unwanted contributions of the marginal frequencies, and so is better for 
our purposes than the unadjusted phi. 'At the same, time,' the adjusted phi provides 
a picture, of better targeting oLCE. 

Adjusted Phi Coefficients with Teacher Judgment of Need for CE. The adjustment was 
also applied to the calculation of phi coefficients when teacher judgment served as the 
definition of need for CE. The distributions of the resulting indexes appear in Table 7-1 1 fqr 
reading and math. The adjustment results in larger index values from those reported in 
Table 7-8 where the coefficients were not adjusted. • ^ 

Point-Biserial Coefficients with Measures of Instructional Services Received. Having 
examined tfcie relationship between the need indicators and CE status, we now tumour 
attention to the relatibnship between these same need indicators and measures of instruc- 
tional services received by the students. To the extent that the intention of CE is tfhat- 
participants receive more or less costly services, targeting iridexes based on. services £nd 
need could be expected to better reflect the spirit of CE programs. An additional advantage 
of using a measure of services received is that even schools without formal CE programs 
could support the calculation of a targeting index (which indicates how well services are* 
focused on the needy). We believe that this is a definite advantage because, even without 
formal CE programs, schools may do, good jobs at targeting supplemental services to 
students t in need of them. 

Two measures of services are Lsed: total exposure (i.e., hours of instruction attended 
during the school year) to reading and math'lnstructions, and total resource cost for reading 
and math instruction. The former, measure was taken from the Student Participation and 
Attendance Records for reading and for math ^SPARM and SPAM, respectively), which 
were completed foyr times during the school year for four' selected weeks. The latteF is a , ( 
cost-weighted composite of many, kinds of 'instructional services and materials, and pro- 
vides a way of quantifying a diverse set of services. . 

The point-biserial correlations for reading and math are presented in. Table 7-,12 for the 
CTBS scores, dichotomized at the v 40th percentile. The corresponding correlations for the 
35th percentile dichotomization' appear in Table 7-13. The index values are not very 
difficult for the two criterion levels, although the 35th percentile cutoff yields slightly higher 
index* values for reading, while the 40th is better for math: Ag^in, the correlations are 
higher for reading than for math, but neither is very high. (The coefficients are lower than 
most of fhe phi coefficients' based on sefection status, but many non-CE schdds' Ijave 
been' included in the tabulation of the point-biserials. We* can conclude that non-CE 
schools do not target services ^o low-achieying students as well as CE schools do.) The best 



Table 7-11 



Cumulative Percentage Distributions by School and Grade: 
Adjusted Phi Coefficient* Between CE Participation 
and Teacher Judgmenjfcf Need for CE/ for Reading and Math 



Reading 



• - Index 


School 


Grade 1 ' 


Grade 2 


Grade 3 


Grade 4 


Oade 5 


Grade 6 


Value 


(N - 205) 


(N*138) 


(N-169) 


(N-172) 


(N= 163) 


(N-153), 


(N«143) 


1 p^q than — 

9 I. CO 9 11 loll 'JJ 


n n 




n fx 

u.o 


n n 


1 7 
\ .z 


n n 


u.u 


. JZ • .ZJ 




^ i 


u.o 


u.o 


1 1 

\ .Z 


n 7 


u.u 


— . ZZ ■ ~ . 1 J 


i n * 


ft n 


1 ft 


i .z 


n n 
U.U 


a 7 


n 7 
U./ 


-.12- - .03 


Z KJ 


in q 


1 ft 
1 .o 


L.j 


Z.J . 


i 

i . j 


i ft 
z.o 


-'02-02 


20 


11 6 


• '3.0 


2.9 


3.1 


2.0 


4.2 


.03- 12 


3.9 


14.5 


3.0 


29 


3.1 


3 3 


6.3 


..13- 22 


59 


14.5 


4 7- 


3 5 


3.7 


3.3 


8.4 


23- 32 


7.3 


15.2 


6 5 


5.2 


V 

4.3 


4.6 


10.5 


.3,3.42 


93 


19.6 


8.3 


8.7 


5.5 


5.9 


13.3 


.43- 52 


13.7 


24.6 


11.8 


11.6 


9 2 


11.1 


16.8 


.S3-.62 » 


20 5 


30.4 


17.2 


16.9 


14.7 * 


15.7 ' 


21.7 


.63.72 


35.6 


39.1 


24.9 - 


— 2S£ 


30.7 


21.o 


30.8 


.73- 82 


49.3 


44.9 


. 33.7 


38.4 


39.9 


31.4 


* 42.6 


.83- 92 


77.1 


61 ?6 


52 1 


50.0 


52.1 


49.0 


55.2 








Math 










Index 


School 


Grade 1 


Grade 2 


Grade 3 


Grade 4 


Grade 5 


Grade 6 


Value 


(N^161) 


JN- 85). ' 


(N- 95) 


(N = 107) 


(N= 108) 


(N-107) 


(N = 102) 


1 p<i<; th^n - 

Lv JJ VI (till >JJ 9 


KJ.Kf 


j . y 


i i 

i . i 




1 Q 

i . y 


0 Q 
yj. y 




- 32 - - 71 

— - i-j 


1 Q 

i .y 


7 1 * 


Z. 1 


1 Q 

l .y 


j . t 


1 Q 

l .y 


<* 


— . zz • — . i j 


j . I 


1 1 ft 

1 1 .0 


J. J 


j.O 


m J.O 


1 Q 
1 .y 




-.12- - .03 


j \j 


1 O.J 




8.4 


fx S 
O.J 


fx ^ 
O.J 


j.y\ 




5.6 


17.6 


10.5 


; 9.3 


7.4 


8,4 


3.9 


. .03-12 


7 5 


' 21.2 


11.6 


12.1 


9.3 


* 11.2 ' 


3.9 


13 .22 * 


99 




15.8 


12.1 . 


' 13 0 


13.1 


7.8 


23 .32 ' ' 


15.5 * 


29.4 


16.8 


14.0 


14.8 


15.9* 


13.7" 


.33-.42 


19.9 


, 36.5 


20.0 


20.6 ' 


16.7 


20.6 


15.7 ' 


.43-. 52 


. 26.1 . 


* 41.2*. 


26.3 


30.8 


23.1 


23M 


21.6 


.53.62 . 


38.5 


45.9 


34.7 


36.4 


28.7 


'28.0 


28.4 


.63.72 


% 52.8 


S0.6 


9 47.4 


47.7 


. 43.5 


34.6 


42.2 


.73.82 


69.6 


60.0 


60.0 


57.0 


56.5* 


49.5 


52.0 


# .83.92 


83.9 


74.1 


71.6 


63.6 


' 64.8 


61.7 * 


60.8 



9 
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102 



1J0 



Table 7-12 



Cumulative Percentage Distributions by School and Grade: 
Point-Biserial Correlations Between Total Exposure to Instruction 
and Achievement Scores Dichotomized at the 40th Percentile 

^ v . 



Index 
Value 



School 
(N-206) 



Grade 1 
(NU182) 



Reading 

Grade 2 
(1^181) 



Grade 3 Grade 4 * Grade 5 Grade 6 
(N = 182) (N =177) (N . 175) (N = 162) 



less than - .63 
-.62- - .53 
- 52 - - 43 
-.42 - - .33 
-.32— 23 



• 22 - ^ 
12 - - 

02- . 02 

03- 12 
13- 22 

23- 32 
33- 42 

.43- 52 
53- 62 
63- 72 
73.82 

.83.92 



13 
03 



0.0 
0.0 
0.0 
' 0.0 
1.9 

7.8 
21 8 
35 0 
69,9 
89 3 

95 1 
99 0- 
99.5 
1000 



0.5 


0.6 


0.5 


«• 1.7 


t).6 


1.2 


1.1 , 


1.7 


05 


v . 2.3. 


0.6 


2.5 


2.7 


V 


0.5 


\ 2.8 * 


2,9 




6.6 


8.3 


4.9 


- \ 6 2 


4.6 


^ 3.7 

r 6.8 


159 


° 16.0 


11.5 


10.2 , 


8.0 


11.7 


28.0 


24!9 ' 


19*2 


16.4 


13 A 


22.2 


' 45 t 6 


34 3 


30 8 


27 1 


25.1 


37.0 


•59.3 


45 3 


374 


32.2 


32 6 


. 43.8 


76 4 


1 602 


48.9 


49.7 


43.4 


54.9' 


90.1 


74.6 4 


64.8 


63} 


60.6 


69 1 


94.0 4 


81.2 


76.4 


71.8 


74.9 


79.6 


96.7 * 


88.4 


"83.0 


\- 797 


82.9 


85.8 


98.9 


90.1 


91.2 


87.0 


91.4 


93 8 


99 5 


92.8 


96.7 


93.2 


96.0 


- 97.5., 


99.5 


97.8 


98.4 


98.3 


^ 7J 


99.4' 


99 5 ' 


99.4 


98.9 


, 99.4 


>8.9 


. 100.0 


100.0 


100.0 - 


98.9 


1OO.0 


^9.4 



Thdex Schoo?" tirade I 1 

Value (N = 206) (N=182) 



Math 

Grade 2 
(N=180) 



Grade 3 
(N = 179) 



'Grade 4 
(N=t 1^6) 



Grade 5 
(N=174) 



Grade 6 
(N=162) 



Less than - .63 


0.0 


0.0 


1.1 


•0.0 


- 62 - 53-1 


> 0.0 


0.5 


2.8 


1.7 


- 52 . - *3 


0.0 


3.8 


4 4 


4.5 


- 42 - - 33 


0.5 


* 60 


9.4 


5.6 


-.32- - 23 


3 9 


12 6 1 


17.8 


1*1 


r 22- -.n 


170 


23 6- 


27.i> 


26.3 


- 12 -,-.03 . 


47.6 


51.6 


48.3 


44 7 


- 02- 02 


, 65.0 * 


59.3 


58.9 


50.8 


03- 12 


89.3 


75.8 


• 73.9 


72.1 


. 13.22 


96 1 


86 8 - 


89.4 


^82 7 


.23- 32 


99 0 


92 9 ' 


95.6 


91 1 


.33. 42 


995 


'97.8 


98.3 


95.5 


43- 52 


100.0 


99.5 ^ 


99.4 


97.2 


53 .62 




100.0 


99.4 


99 4 


.63-72 






100.0 


99.4 


7<3- 82 








99.4 


83-92 








.99.4 



0.6 
1.7 
4.5 
8.0 
15 9 

28.4 
46.6 
55.7 
71.0 
83.5 

89 8 
96.0 . 
97.2 
98 } 
99.4 
100.0 



0.0 
1.1 
2 9 
- *8.6 
17.8 

31.0" 

46.6 

59.«2 

71.8 

79.3 

87.9 
93.1 
96.0 
98.9 
99.4 
99.4 
100.0 



1.2 
1.9 
3.7 
6.8 
16 7 

32 J 
53.7 
^1.7 
75 9. 
86.4 

93.8 
■ 97.5 
- 98.1 
98.8 
99.4 
100.0 



f _ 
Note The indexes in the middle set of rows of each subtable (-.22 to +.22) have values that are (approxi- 
mately) not stgmficajntly different from zero-or chance targeting 6f services . # 

t 
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< * . Table 7-13 

K Cumulative Percentage Distributions by School anc( Gratje: 
Point- Biserial Correlations Between Total ExpoStire to distinction 
and Achievement Scores Dichotomized at the 35th Percentile 









' c Reading 






• * 






School 


Grade 1 


Grade 2 ■ 


Grade 3 


^Grade 4 


Grade 5 


Grade 6 


Value <£• 


(N-206) 


H(N«182) 


(N-179) 


(N-18<% 


(N-176) 


(N-174) 


*>H-162) 


Less than - .63 


* 0.0 


0.5 


1.7 


0.6 ' 

♦ 


1.1 


0.6 




- 62 - - .53 


0.0 


1.1 


A 2.8 


1.1 


1.7 


• 0.6 * 


3.1 . 


- 52 - - 43 


0.0 


2.7 


4.5 - 


2.2 > 


' 2.3 


3.4 


3.7 ' 


-.42 - -.33 


0.0 


8.2 


7.3 




5.7 


5.2 


6.2 


-.32 - - 23 


1.9 


14.8 


12.8 


11.7 * 


10.8 


8.6 


13.0 


- .22 - - 13 


- 

7.8 


31.3 


• 

25.1 


20.0 v 


15.3/ 


16.1 


223 


-.12 - 03 


24.8 


• 50.0 


36.3 k 


29.4 


28.4 


25.9 




-.02- .02 


38.8/ 


57.7 


43.6 


*35.6 


34.7 


32.8 


45.7 


.03-. 1 2 


. 73.8 


77.5 - 


60.3 * 


51.7 


.50.6 


44.3 


56.8 


.13-.22 


91.3 


86.8 


75.4 ' * 


65.0 


60.8 . 


* >2.1 


69.1 


.23-. 3 2 


94.7 


93.4 


85.5 


7-1 ■ 


t\j. d 


/ J.U 


on i 


\33-.42 


"$9.0 


96.2 


88.3 


82.2 


79.0 


83.3 


87.7 


.43-.52 


100.0 


99.5 


92.2 


90.0 


84.1' 


.90.8 


•92.6 


53.62 




99.5 


96.6 


96.7 T 


92.6 - 


94.3 


m 98.8 * 


.63-. 72 * 




*99.5 


98.3 • 


98.3 


95.5* 


%.6 


99.4 


73-.82 




rog.o 


99.4 


98.9 


98.9 


97.7 


99.4 • . 








100.0 


98.9 


100.0 


99.4 


99.4 








Matfi 










Index 


School 


Grade 1 


Grade 2 ~ 


Grade 3 


Grade 4 


Grade 5 


4 Grade 6 


value 


(N ■ 206) 


(N * 182) 

X 


(N» 180) 


(N-179)* 


(N> 176) 


(N« 174) 


> (N-162) 


Less than -.63 


0.0 


0.0* 


1.1 


0.0 - 


0.6 


0.0 


0.6 


-.62 - -.53 


0.0 


0.0 


1.7 


0.6 * 


' 1.1 


O'.fr 


1.2 


-.52 - -.43 


' 0.0 


% 2.7 


*' 5.0 


«3.4 ' . 


- 3.4 


4.0 


o 2.5 


-.42 - -.33 


. 1» 


4.4 


9.4 


6.7 . 


7.4, 


8.6 


' 4.3 


-.32 - -.23 - 

— 


1.5' 


'9.9 


16.7 


13.4 - 
* 


< 13.6 


17.8 


12.J 


-.22 - -.13 


15.0 


23.1 * 


29.4 


23.5 


28.4 * 


♦ ' 31,0 


29.0 


-.12 - -.03 * 


^46.1 


46.7 


. 46.7 « 


46.9 2 


46.6 


50.0 


) 54 3 ' 


-.02*-.02 


^ 58.7 


S5.5 


56.7 


55.9 


• 56.3 


' 56?3 


m': 2 


.03-. 12 * 


^86.9 


74.7 


76.1 


70.9 


73.9 


- 7l6 


■ 79.0' 

1 


.13.22 


95.1 


86.8 


91.7 


84.9. 


'85.8 


* 79.3 


88.3 


23.32 „ 


99.5 


92.3 


95.0 - 


90.5 


92.0 


88?5 


95.1 


.33-.42 


99.5 


».3 


98.3 


. 93.9 * 


96.6 


92.5* 


98.1 < 


.43-.52 


100.0 


98.9 


98.9 


97.2 


98.3 


96.0 


98.] 


.S3-.62 




100.0 


k 99.4 


99.4 


. 98.9 


98.9 


98.8 


.63-72 






100.0 


99.4 


100.0 


100.0 


99.4 , 


.73-.82 








99.4 






100.0 


.S3-.92 








99.4 * 
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cutoff for reading (35th percentile for schools) yields 38.8 percent no n-posit i ve^ index - 
values, and for. math (40th percentile for schools) it yiefds 6$.0 j>efcent non-positive index* 
values Table 7-14 ^contains the cumulative distributions for reading and math of the 
indexes created by correlating teacher judgment of student' need for CE with exposure to 
instruction '(reading rped with reading instruction, math nee'd with math instruction). The 
percentage of non-positive correlations^ 34.0 forre,adirig and 56.8 for math-not -much 
better than for the (Irevious-criterion. . * \ 



Finally, the s^me sefof CfBS dicfiotompus scores and the teacher judgment variable are 
^correlated with the corresponding resource cost measures in Tables 7-15 to 7-17. The v 
correlations based on resource cost are larger-than those based on services. Again, there is 
littjft difference between the correlations bbtainec^for the two percentile cutoffs, -and again * 
the,correlations for reading are considerably higher than those for math. Teacher judgment 
is a better correlate of level of service th^n either achievement-score dichotomy;, jyst as it 19 ' 
a better correlate of CE ^ejection (5ee-previous remarks on limits to the Validity of indexes - 
using this measure). x t * *' 



INDEXES BASED ON CORRELATIONS WITH CTBi PERCENTILE SCORES ? 

In the previous sections, several indexes have been '-derived from. correlations bef^veen* 
dichotomized CTBS scores and 'CE status or measure? of CE services received, li is- also qf 
interest to examine indexes that are b^sed on*correlatibns involving the CH55 scores ' 
themselves. The form of the achievement scores used for these anaJyses islhe percentile, as 
we expected "that the percentile 'scores would have more appropriate distributions within . 
•grades and within schools^he indexes to be described were created by correlating, foe 
both reading and math, the percentile scores yvithCE status, with total exposure to instruc- 
tion, and with resource cost of instruction^ Because the CE-status variable, is dichotomqus, 
the correlation between it and the CTBS score is a point-bise'rial correlation. The variables 
reflecting the amount of instructional sfce^es received, however, are continuous and * 
quantitative variables; thus* the correlations with the CTBS scores are Pearson .product^ 
moment coefficients. - • • » v 4 

The' correlations between the^CTBS scores and CE status are the /indexes in Table 7-18. T^e* 
table contains the cumulative frequencies -for the indexes "by grade and by, school, Because 
low percentile scores* (negative when' standardized) Ire associated with CE partfeipatiori 
(positive when standardized) „ when good tatg^Ung 'occurs, good targeting would be 
indicated by negative correction coefficients. |n orcle* to make the indexes of this section 
comparabale .to those, already studied, and in line with intuition, We have reflected all thfr 
coefficients (changed their signs) in this section. . j . • 

Except for the first grade, the correlations forreadtng, are overwhelmingly positive (i!e., 
indicating good targeting), with onJy^.6 percent to 3-6 percent of the values ^gative. The" 
median value at the school level \% betvyeen".35 ancK.45, however^ indicating. only 
moderately 'effective targeting., As wirt^th^ previous index,"*the math vafues indicate^ 
somewhat poorer targeting than thos^for reading.. 4 , / , ?• 



Table 7-14 



Cumulative Percentage, Distributions by School and Grade: 
Point-Biserial Correlations Between Total Exposure to Instruction 
and Teacher Judgment of Student Need for CE 











Reading 


€ 


>»• 








Index 
Value 


Scheol 


Grade 1 

(f\ » 1 /5J 


Grade 2 

/Kl 4 01\ 


Grade 3 
(N = 184) 


Grade 4 t 

. (N« 177) r 


Grade 5 
(N-173) 


Grade 6 
(N-159) 




Less than - .63 
^.67 53 . 
p -.52 - -.43 - 
>ie -.42. -.33 
32 - -..23 


^ 0.0 

- o.(f ^ 

* ■ 0.0 
0.5 
1.9 
/ 


•23 * 
v > 2.3 
4.6 
13.1 
52.9 


' 0.6 
0.6 
2.8 
8.3 
15.5 


0.5 
0.5 
2.7 
4.9 
7.6 


0.0 
0.6 
2.8 
5.1, 
6.8 ' 


0.0 
- 0.0 
2,-3 
5.8 
9.2 


1.3 
1.3 
3.8 
5.0 
15.1 


• 


- 22 - - 11 
-.12-- -.03 , 

- .02- 02 ' 
03-. 12 
13-.22 


Q 7 

' 23.8 
'340 
63.6 \ * 
83.0 , " 


77 1 

^49.7 
56.0'' 
72.6 
82.9 


23. o 
34.8 
40 3 
• 53.6" 
65.2 


15.2 
28.8 
32.1 ~ 
40.8 
54.3 


14.7 
26.0 ' 
£.0,0 
41.2 
50.3 


15.0 * 
24.9 • 

7C\ fx 

42 8 
54.3 


21.4 
3*0 

j4 1 C 

51.6 
58.5 




23-. 32 

.33-.42 • • 
.43-. 52 

'.53-.62 * 
* .63* 72 
.73-.82 

T83-.92 

* 


91.3 
96.6 

-97 K 
98.5 

100.0^ 

♦ * 


88.6 
92 6 
96'0 * 
96.0 
97.7 . 
98.3 

100.0 ^ 


. 7*5 
79.0 
83.4 
90.6. 
94.5 ' 
98.3 
^vlOO.O * 


65.2 

75.5" 

81.6 

88.6 

92.9 • 

95.7 

97.8 ' 


61.0 
69.5 
76.8 ' 
, -80.2 
85.3 
qi n 
96.6 


64.2 
71.1 
79.2 
85.5 
90.2 

Q7 fx 

97.7 


• 66.0' 
74.8 

' 81.8 
86.8 
90.6 

yj.U 

99.4. 


1 

■4 








Math 




t 








• Index 


School 

(iN = ZlKjJ 


•Grade 1 
(N = 1 73) 


Grade 2 
(N * 172) 


Graded 
(N» 173) 


Grade 4 
(N-173) 


Grade "5 
(N-171) 


Grade 6 
(N»158) 




Less-than - .63 - 

- .62 - - .53 
'-.52 ---.43 

-.42- -.33 
.-•31* -.23 * 


0.0 
0.0 
. -0T5 
1.5 
58 


1.2 

'2.3 - 

ri.o 

723.7 


1.7* 
2.3 
2.9 
6.4 
16.f* 


1.7 
2.9 

-4.6 
8.7 

17.3 


0.Q 
1.7 
3.5 
11.6 
15.6 


1.2 
2.3 
4.7 ' 
9.9 
16.4 


0.0 
2.5 
« 7.0 

10.1 . 

18.4 




-.22, -.13' 
- -12 - — 03 

• - .02-.02 

. .03-. 12 
.13.22 


1^.4 . 
. 41.7 / 
' S6.8 / 
73 J 
r * 87.4 


/ 38.2 • 
54.9 
61.8 
76.9 
86.7 


3.2.0 
46.5 
51.7 
, *68.0 
79.7 


28,3 ** 
42.8 • 
52.0 
67.1 
, 78.0 T 


^2fr.Q 
40.5 
47.4" 
61.3 
' 76.3 


32.2 
45.6 . 
54.4 
64.9 
77.2 


• 31.0 * 
46.2 

* 57.0 
68.4 r 
76.6 




. .23- 3? ~ 
.33-.42 . 
.4J-.52 
.53- 62, 
.M-.72 * 
9 .73.82 
.83->92 

— €t — — 


95.U 
' 97.1 
98.5 

v 99.5, ' 

100.0 
• > 


91.9 
, 95.4 
97.1 
. 99.4 
99.4 
100.0 


84.9 • 

AM- / 
91.9 / ' 
94.8" ' 
97.7 
r99.4 
99.4 


85.0 
89.0 
94.2 
96.5 
98.3 
98.8 
99.4 


80.9 
86.1 
89.6 
96.0 
. 97.7 
98.8 
99.4 . 


82.5 

85.4 • 

90.6 

94.2 . 

96.5 

96.5 

98.8 


87.3 
90.5 
SI. 8 
94.9 
96.2 
98.1 
98.7 



Note- The indexes in the middle set of rows of each subtable(- .22 to +.22) have values that are (approxi- 
noaldyli^sjgnificantly different from zero^or chance.targeting of services. ' 
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Table 7-15 



Cumulative Percentage Distributions by School and Grade: 
Point-Biserial Correlations Between Resource Cost of Instruction 
and Achievement Scores Dichotomized* at the 40th Percentile 



v . > Reading 

Index School Grade 1 Grade 2 * 

Value (N=» 206) (N ^182) - (N = 181) 



Grade 3 Grade 4 Grade 5 
(N = 180) (N=177) (N-175) 



Grade 6 
(N«161) 



Less than -..63 

- 62 • - 53 1 
-^52 • - 43 
-.42 • - 33 
-.32 - -'23 

- 22 - - 13 

- 12 - - 03 

- 02- 02 
03- 12 
.13-22 

23- 32 
' 33- 42 ' 
43-. 52 
*53- 62 
.63-. 72 
.73-62 
. 83-92 



0.0 


0.0 


J).6 


0.6 


0.0 


n n 

u.u 


1 7 


0.6 


0.5 


1 6 


2 8 


2.2 


0.5 


5 5 


, 5,0 


3.3 


0.5 


7 1 


5.5 


6.1 


2 4 


170. 


7.2 


8.8 


6.3 


33.5 


13.3 


10.6 


102 


42 9 


15 5 


11 7 


24 3 


58.8 


23.2 


21 0 


50.0 


76 4 


. 35 4 


28.9 


81.1 


85.7 


43.1 


39 4 


94.2 


92.9, ' 


56.4 


52.2 


98.5 


97.8 


N 73.5 


66.7 


99.5 


. 99.5 


89.5 . 


83.9 


idb o 


•99 5 


95.0 


93.3 




99.5 


98.9 


97.8 ' 




J 00.0 


100.0 


* 99.4 




U- 











Math 





0.0 


0.0 


0.6 


0.0 


" 0.0 


0.6 


1.4 


♦ 0.6 


1.2 


1.1 


2.3 


3.7 


5.6 


2.9 


5.6 


7.9 


5.2 


9.9 


13,6 


7.4 


14.3 


169 


12.0 


180 


20.9 


17.7 


26.1 


30.5 


29.7 


404 


42 4 


46.3 


58.4 


56.5 


64.0 1 


v * 72.7 


69.5 


77.1 


85.1 


86.4 


89.1 


*.2 


97.7 


94.9 


* 98.1 


98.9 


97 7 


99.4 


98 9 


98.9 


100.0 



{ Index 
Value 



School 
(N = 206) 



Grade 1 
(N = 182) 



Grade 2 
(NU180) 



Grade \ 
(N = 178r 



Grade 4 
(KU176) 



Grade 5 
(NU174) 



Grade 6 
(N = 161) 



1 Less than - 63 

- 62 - - 53 

- 52 - - 43~ 

- 42 - - 33 

- 32 - - 23 

-.22.- - 13 

- 12 -* - 03 
-.02-.02 

.03- 12 
J3-.22 , 

23-* 32 

33-. 42 
. 43-. 52 ' . 

53- 62 9 
.63- 72 
.73- 82 '* 

83- 92* ^ 



00 
00 
0.0 
0.5 
1.0 

49' 
17.0 
32 5- 
55 3 
81 1 

95 6 

98 1 

99 5 
100.0 



0.0 
• 0.Q 
1.1 
1 6 
6.6 

13.2 
28.6 
39 6 
5,8 2* 
76.4 



89,0 
94 5 
98 9 
99.*5 
1000 



- 00 
0.6 
0.6 
5,0 
13.3 

18.9 
, 30.6 
37.8* 
52.2 
71.7 

81,1 
91 1 

96.7 
98.3 
99 4 

ioo.a 



0.6 
1.7 
2.2 
4.5 
10.7 

20.8 
29.2 
33.7 
46.6 
59.0 

77 5 
88.8 
92.1 
95.5 
98.3 
98.9 
98.9 



o.o . 


0.0 • 


0.0 


00 


1.1 


0.6 


1 7 


2.3 


1.2 


5 7 


4.6 


3.1 


9.1 


10.9 


8.1 


12,5 


16.7 


17.4 


25.0 


27.0 


.24.2 


35 2 


35.6 


30.4 


49.4 


48^9 


47,2* 


67A 1 


61.5 


63.4 


78.4 


71.8 . 


77.0 


88 6 K 


82.2 


90.1 


94.3 


89.7 


95.7 


97.7 


97.1 


98.8 


100.0 


99 4* 


100.0 




100.0 





. Note The indexes in the middle set of rows of each subfable (-.22 to +.22) have values.that are •(approxi- 
matelyf not significantly different from zero-or chance targeting of service costs. - 
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- Table 7-16 



Cumulative Percentage Distributions by School and Grade: 
Point-Biserial Correlations Between Resource Cost of Instruction 
and Achievement Scores Dichotomized at, the 35th Percentile 



m 






Rpadinff 






















index * 


School 




vfiautr a 




'Oracle ^ , 




Vila lie O 


Value 


(N - 206) 


(N- 182) 


(N-179) 


(N-|78) , 


(N-176) 


(N- 174) 


(N - 161) 


* ■ — j 

Less toan - .63 


0.0 


0.0 


* 1.1 


0.6 


0.0 


0.0 


0.6 


- .62 - - 53 


0.0 


0.0 


2.2 


0.6 


0.0 


o.o 


- 0.6 


- 52 - -.43 


0.0 


0 5 


2.8 


1.7 


0.0 


0.6 


0.6 


- 42 - -.33 


0.5 


4 4 


3.4 


2.8 


0.6 


2,3 


3.1 


-.32 - - .23 


1.5 


10.4 


3.9 


5.1 " 


2.8 


2.3 


5.6 


- 22*- - 13 


2 4 


18 7 


8.9 


79 


1A 


5.2# 


9.9 


-.12 - - 03 


6 3 


36.8 


13.4 


10.1 


14.2 


8.(f^ 


13.7 


-.02- .02 


9 2 


45 6 


15 6 


13.5 


170 


11.5' 


19.9 


03-. 12 


24.8 


66.5 


22.3 


19.7 


21.6 


16.7 


27.3 


1 1 11 
i j- 


q 

jj." 


on 7 






in 7 


27/) 


Jb.b 


23. 32 


81.6 


89 0 1 


46.4 


39.3 


42.6 




N 

57.8 


33- 42 


93.7 


95.1 


i 60.9 


50.0 


56.8 % 


58.6 


73.9 


.43-.S2' 


98.1 


98.4 * 


78.2 


66.9 


69.3 


78.7 


83.2 


.S3-.62 - 


100.0 


99.5 


87 7 




* 84.1 


89.1 


94.4 


.63-.72 




99.5 


93.3 




94.3 


94.8 


97.5 


.73-.82 




1000 


97.8 


97.2 


96.6 


97.1 


98.8 


83-.92 






99.4 


'99.4 


98.3 


98.9 


99.4 




- 




Math 
trial II 










^ Index 


Srhnnl 


Grade 1 


Grade 2 


C,rat\f> 1 
viiauc J 


VJiallc *♦ 


ViiaUCr D 


Oracle o 


Value > 

■ 


(N - 206) 


(N - 182) 


(N-180) 


(N-178) 


(N-176) (f 


(N-174) 


(N- 161) 


Less than - .63 


0.0 


0.0 


0.0 


0.6 


0.0 


, 0.0 


0.0 


-.62 - -.53 


_ 0.0 


0.0 


0.6 


1.1 


0.0 


0.6 


0.6 


-.52 • -.43 


0,0 


0.5 


1.1 


1.1 


.1.7 


1.7 


1.2 


-.42 • -.33 


0.5 


1.6 


5.0 


3.4 


5.1 


3.4 


4.3 


-.32 - -.23 


1,0 


5.5 


9,4* 


8.4 


9.1 


11.5 


7.5 


-:22 - -.13 


4.9 


13.2 


20.6 


18.5 


14.8 


18.4 


» 

15.5 


-.12 - -.03 . 


17.0 


30.2 


30.6 


29.2 


25.6 


27.0 


24.8 


o - .02-.02 * 


30.1 


40.7 


39.4* 


33.7 


37.5 


35.6 


28.6 . 


, .03-.12 • 


52.4 


w 61.5 


51.7 


46.1' * 


51.7 


47.1 


46.6 


.13-.22 


79.6 


75.3 


67.8 


60.1, 


67.6 


61.5 . 


, 64.0 


.23-.32 


95. r 


84.1 


81.7 


78.1 


80.1 


71.8 


78.3 


.>3-.42 . 


98.1 


91.2 


90.6 4 


t 86.5 


' 88.6 


81.6 


87.6 


.43-.S2 


99.0 % 


97.8 


94.4 


92.1 


96.6- 


89.1 


95.0 


.S3-.62 


100,0 


98.9 


98.9 


96.1 


98.3 


97.1 


* 98.8 


.63-72 




99.5 


99.4 


97.8 


99.4 


100.0 


99.4 


.73-.82 




99.5 X 


100.0 


98.9 


100.0 




100.0 


83-.92 




99.5 




98.9 

























Note The indexes in the middle set of tows in each subtable (-.22 to +.22) have values that are (approxi- 
mately) not signfiaantly different from zero— or chance targeting of service costs. 
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Table 7-17 . -„ 

Cumulative Percentage Distributions by School and Grade: 
s • * Point-Biseriai Correlations Between Resource Cost o^lnstruction 

and teacher Judgment of, Student Need for CE 

. ,* ? 

Reading * * 

Index School Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 

Value (N-206) (N-«175) (N-181) (N*182) (N-177) (N-173) (N-158) 



7" 



Less than -.63 


0.0 


0.6 


1.1 A 


1.6 


0.0 


0.0 


1.9 


-.62 - -.53 


0.0 


0.6 


'1.1 *' 


1.6 


0.0 


0.6 


1.9 


-.52- - 43 


0.0 


_ 2 9 * 


2.2 


2.2 


, 06 


1.2 / 


1.9 


- .42 - - 33 


0.0 


69 * 


*;§ A" 


5.5 


2.8 


1.7 


3.2 


r -.32 - -.23 „ . , 


00 


8.6 


5.0^ 


5.5 


4.0 


4.0 


5.7 


— - -.13 


0 5 


1 3.7 


7.2 


6.6 


4.0?i ' 


6.4 


8.9 


-.12 - -.03 


34 


22.9 


* 12.2 


9.3 


'6.9'& 


8.7 


12.0 


- .02-.02 


' 63 


26 3 


138 • 


9.9 


10;7jP 


11.6 


15.2 


.03-. 12 


. 15.0 


33.7 


17 1 


14.3 


16.9^ 


; 16.2 


22.2 


13- 22 , 


• 31.7 


43.4 


27.1 


19.2 


19.2 ^ 


vrA 


26.6 


23.32 / 


0 

S1.5 


49.1 


35.9 


36.2 


v 28.8 


30.1 


34.2 - 


33-.42 f 


69.9 


62.9 


45.9 


37.9 


* 37.9 


39.9 


46.2 


.43-.S2 \ 


85 4: 


72.6 


56.4 


48.9 


50.3 


54.3 


.56.3 


.S3-.62 


95.1 


78.9 


65.2 


62.1 


62.7 


- 71.1 


69.0 


.63-.72 


99.0 « 


87 4 


78.5 


75.8 


73.4 


80.3 


77.2 


,73-.82 « 


-1O0.0 


93.1 . 


90.1 


89.0 


88.1 


91.3 


89.2 


.83-.92 




98.3- 


98.3 


94.5 


95.5 . 


97.1 


98.1 








Math 










Index 


Schbol ' 


Grade 1 


Grade 2 


Grade 3 


Crade 4^ 


Grade 5 


Grade 6 


Value 


(N=206) 


(N«173) k 


' (N*172) 


(N = 172) 


(N=*173) 


(N=*171) 


(N = 158) 



Less than ^ 63 


00 


0.0 • 


0.6 


0.0 


0.6 


9 1.8 


' 0.6 


-.62- -.53 


0.0 


0.0 


2.3 


1.2 


- 2.3 


2.3 


1.9 


-.52 - - .43 


0.0 


* 0.6 


2.3 


2.9 ' 


v 4.0 > 


5.3 




-.42 - -.33 


0.5 . 


4k) 


2.9 


, 5,8 


5.8 


7.6 


4.4 


-.32- - 23 


1.5 


8.7 


I 


. ' 8.1 


9.2 


14,0 


6.3 *' v 


- 22 - -.13 


4.4 


1,2.1 




12.8 


13.3 


'17.5 


12.7 


.-.12 - -.03 


. 9.7 


22.5 *. 


24.4 


20.3 


20.2 > 


25.1 


22.8 


.-.02-.02 


20.4 


"32.9:- 


28.5 


■24.4 


y 23.1 


32.2 


2^.2 


.03-. 12 


■ 35.9 


48.0 


44.2 


34.9 


'32.9 


39.8 


s34.2 


, .13-.22 


56.3 


56.1 


54.7 . 


44.8 


, 42.2 


45.6* 


46.2 , 


.23.32* 


73.8 


- 6§.2 • 


62.8 ! 


57.0 


56.1 


57.3 


52.5 


J3-.42 . 


85.4 


76.3 


• 69.8 


70.3 


H 67.1 o 


64.9 


- -62.7 


.43-.S2 


92.7 


. 82.1 


* 78.5 


. 77.9 


7&Q 


76.0 


72'.8 


53-.62*s^ 


58.1 * 


88.4 


85.5 


86.6 - 


85.0 


'8h9 


81*0 


63-: 72 


• 99.0 


92.5 


4 91.3 


93.6 


91.9 


89.5 


89.9 


.73-.82 \ 


99.5 


* 97.1 . 


98.3 


96.5 


97.1 


94.2 


94.3 


83-.92 . 

f 


100.0 


98.3 * 


98.8 


97.7 


99.4 


98.8 


98.7 



Note The -indexes ir^?nV«iddle set of Vows in each subtable (-.22 to +.22) have values that are '(approxi- 
mately) not significantly different from zero— or chance targeting of service costs. 
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Table 7-18 

t 

Cumulative Percentage Distributions by School and Gr^de: 
Point-Biserial Coefficents Between CE Participation and 
, , Achievement Percentile Scores 









Reading 






• 




index 


School 


Grade 1 


Grade 2 


Grade 3 


Grade 4 


Grade 5 


Grade 6 


Value 


(N = 205) 


(N- 143) 
— 


(N- 169) 


(N-172) 


(N» 164) 


(N» 156) 


(N « 144) 


Less than - .43 


0.0 


• 0.7 


0.0 


0.0 


0.0 


0.0 


0.0 


• -.42 --.33 


0.0 


2.1 


0.0 


0.0 


0.0 


0.0" 


0.0 


-.32 - -.23 


0,0 


3.5 


0.0 


0.6 


0.0 


0.0 


0.0 


-.22 - -.13 


00 


8.4 


1.2 


1.7 


0.0 


0.6 


0.0 


- . 1 1 - — .U J 


1 n 


16.8 


3.6 


3.5 


0.6 


0.6 


2.1 


- .02-.02 


2.0 


24.5 


4.1 


4.7 


1.8 


13 


2.1 


,Uj-. I Z 


S Q 


4b. 1 


9.5 


5.8 


7.3 


3.2 


7.6 


.13-. 22 


20.0 


69.9 


17.8 


13.4 


18.3 


9.0 


22.9 


.23-. 32 


41.5 


83.9 


• 27.2 


25.0 


31.1 


19.9 


39.6 


.33-.42 


66.3 


• 94.4 


41.4 


37.8 


47.6 


40.4 


59.7 


.43-.52 


91.'2 A 


. 97.9 


54.4 


57.6 


70.1 


63.5 


73.6 


.53-.62 • 


98.5 


98.6 


74.0 


80.2 


•84.1 


79.5 


88.2 


.63-.72 


9$.5 


98.6 


91.7 


91 '.3~ 


93.3 


94.9 


95.8 . 


JJ-.82 


100.0 




Qft ft 


Qft 1 1 


Q7 n 


Q& 1 


Q7 Q 


.83-. 92 




99.3 


QQ d 


Qft 3 


QQ A 


Qft 7 

JO. / 


^ Qft A 








Math 










Index 


School 


Grade 1 


Grade 2 


Grade 3 


Grade 4 


• 

Grade 5 


GradMi • 


Value 


(N-162) 


(N = 86) 


(N = %) 


fc (N=109) 


(N* 108) 


(N m 108) 


(N = ufe) 


tess'than - .43 


0.0 


1.2 


1.2 


0.0 * 


0.0 


0.0 


0.0 


-.42 - -.33 


0.0 


2.3 


•* 1.0 


4.6 


0.9 


0.0 


0.0 


- 32 - - 23 


r 0.0 


3.5 . 


1.0 


5.5 


" 1.9 


0.0 


0.0 


-.22 - - 13 


1.9 


11.6 


3.1 


•> 

'7.3 * 


- 3.7 


0.0 


1.0 


-.12 - -.03 


ft ft 


17 4 
1 / .*? 


j. \ 


in 1 


1 7 


U.U 


3.0 


-.02-.02 ' ' 


10.5 


25.6 


8.3 


11.0 


7.4 


0.9 


4.8 


.03-. 12 * 


24.7 


39.5 - 


24.0 


16.5 


13.0 


11.1 


14.3 


.13-.22 


45.3 


59.3 


42.7 


25.7 


• 36.1 


23.1 


40.0 


.23-. 32 * , 


77 8 


83.7 


55.2 * 


51.4 


• 53.7 


42.6 


65.7 


.33-.42 * 


* 91.2 


88.4 


77.1 


68.8 


80.6 


68.5 


80.0 


.43-. 52 


93.7 


93.0 


90.6 


81.7 ' 


89.8 


81.5 


87.6 


.53-.62 


98.8 


98.8 


96.9 


89.0 


96.3 


90.7 


96.2 


63-. 72 


99.4* 


100.0 • 


97.9 » 


95.4 


98.1 


' 98.1 


99.0 


* .73'.82 


160.0 




99.0 


96.3 


100.0 


99.1 


100.0 


.83-.92 






\00.0 


97.2 1 




• 100.0. . 





Note All coefficients have reversed signs, so high indexes indicate good' targeting. The indexes in the middle 
'set of rows in each subtable (.-,22 to + .22) have values that are (approximately) not significantly different 
from zero— or chahce targeting. 
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Vrhe indexes described in Table 7-19 are the correlations between the same reading and 
-math subtest scores and the total exposure to reading and math instruction. The positive 
correlations are again indicators of relatively 'good targeting. Although this measure of 
instructional service cannot be analyzed into CE and non-CE components,, it is of interest to 
examine the amount of agreement between achievement level and amount of reading and 
math instruction to which the student is exposed, as. an indicator of the degree to which 
need is associated with amount of services delivered to the student. The values of these 
indexes have surprisingly negative distributions. .For reading, the median values for all 
grades and for the entire school are lower (i.e., worse) than .20; for math they are all lower 
than -.05. These median values indicate that there is precious little association between - 
percentile scores (need) and numbers of hours of exposure to reading and math. Such a 
small overall -effect may reflect the grossness of the service measure; every student in a 
school receives approximately the same number of hours, so hours may be an insensitive 
measure. Instead, perhaps, we should examine the quality of the services received. 

The resource cost measures represent an attempt to get at the quality of the services by 
defining them according to their relative costs. For example, an hqur of individual instruc- 
tion with a special teacher is costed at a higher level than an hour in a classroom with a 
teaching assistant Table 7-20 presents the correlations between the CTBS scores and the 
resource-cost measures. The indexes show stronger (i.e., more positive) relationships than ' 
those based on hours of services received, demonstrating that targeting is going on in the 
manner expected (if it weren't, improving the variables would make the indexes lower). •' 
The medians for the indexes are more positive, about + .40 for reading for most gVades and 
< from^about +.10 to + .20 for math. The values still do not indicate a strong relationship,' 
however, with most correlations falling at or below the level required for positive statistical 
significance at the .05 level. 

In summary, although it seemed worthwhile to examine indexes based ^on percentile 
scores, these indexes are only slightty more sensitive to targeting than the indexes con- 
sidered previously. We should not be very surprised at this, because the dichotomies are 
merely.coarse groupings of the percentiles, so the correlations based on them are expected 
to be smaller. Both percentiles and dichotomies are based on national norms, while good « 
targeting of services, to students ought to be based on a finer tuning within each school. 
Thus, neither the indexes of this section nor those of previous sections are finely tuned to 
the relative needs of students withm a given school. * 

INDEXES BASED ON ACHIEVEMENT RANKS WITHIN SCHOOLS 

Perhaps the most stringent, concept of targeting is that,, given a set of N students, of whom 
only n can be selected for compensatory services, the optimal allocation of services occurs 
when the n -most needy of the set are selected. Each of the various subsettmgs of the n 
students into two groups (selected and not selected) can be given a value"that reflects how 
well the subset focuses C£ on the most needy and not on the least needy. A statistic 
suggested by Albert Beaton of the Educational Testing Service is based on this definition of 
optimal targeting/ It assigns a value to any subset (of size n) of the entire population of size 
derived from the number of subsets of the same size that would be less targeted. For 





Table 7-19 



Cumulative Percentage Distributions by School and Grade: 
Product-Moment Cofefficients Between Total Exposure to 
Instruction and Achievement Percentile Scores 



.r 






Reading 










Index 




VJiMUC 1 


UidQC ** 


r rar j« i 
Vjraae j 


uraae 4 


uraae j 


Grade 0 


Value ' 


(N-206) 


(N - 183) 


(N-183) 


(N - 184) 


(N-180) 


(N-178) 


(N - 162) 


- ■ - 

lessthan - .43 


0.0 


• 

1.1 


4.4 ;* 


1.1 


3.3 


1.7 


3.7 


.- 42 - -.33 


0.5 


7.7 


9.3 


6.0 


5.6 


3.9 


7.4 


s -.32 - -.23 


3.4 


17.5 


18.6 


14.1 


9.4 


8.4 


11.7 


_ 11 ii 


Q 7 


in f\ 

29.0 


- 

25.7 


' 21.2 


15.0 


18.0 


c 

24.7 


-.12 - -.03 


27.7 


45:9 


37.7 


29.3 


ZO. J 


9Q 1 


17 7 

3/./ 


* -.02-.02 


, 42.7 


55.7 


45.4 


35.3 


34.4 


33.7 


44.4 


03-. 12 


71 4 


7^ 1 




AR. A 


X7 Q 

4/.o 


An q 
4/.o 


57.4 


.13-.22 ' 


87.9 


88.5 

\ 


♦ 68.9 

< 


63.6 


59.4 


60.7 


67.9 


.23-. 32 






v/.u 


71 Q 
J Z.iJ 


DO. I 


7>l 7 

74.7 


77.2 


.33-.42 


\ 98.5 


97.3 te 


85.2 . 


82.6 


77.8 


81.5 


82.1 


.43-. 52 


V 99.5 


97.8 


88.0 / 


38.6 


85.0. 


88.2 


90.1 


.53-.62 


^ 99.5 


97.8 


91.8 


95.1 ' 


92.8 


92.7 


'95> 


A1 71 
•Oj-./Z 


1 00.0 


99.5 


99.5 


97.3 


* * 97.8 


97.2 


98.1 


73-.82 




joao^ 


99.5 \ 


97.8 


99.4 


98.9 


99.4 


.83-.92 






' 100.0 


97.8 . 


99.4 


100.0 


100.0 






i 


Math 












School 


Grade 1 - 


VJiauc * 


ViiaQC -J 


Kit due ** 


uraae 3 


ijraae o v 


Value 


(N-206) 


(N-183) 


(N-183) 


(N-183) . 


(N-180) 


(N-177) 


(N - 162) 


Less than - .43 


0.0 


3.3 


6.6 


5.5 


4.4 


3.4 


3.7 


-.42*- -.33 


1.0 


4.9 


13.7 


9.8 


11.3 


7.9 * 


7.4 


-.32 - -.23 


5.8 

* 


13.7 


20.2 " 


15.3 


17.8 


'l6^ 


18.5 


-.ZZ - - .13 


18.0 


30.1 


31.7 


30.6 


27.8 


33.3 


36.4 


-.12 - -.03 


46.6 


55.2 


47.0 : 


47.0 


48.9 ' 


50.3 


59.3.. 


-.02-.02 


64.6 


60.1 


60.1 


54.1 


59.4 


. - 56.5 


' 67.3 


.03-. 12 


87.9 


.76.0 


75.4 r 


69.9 * 


70.6 


• 68.9 


80.2 


.13-. 22 


i 95.1 


86.9 • 


85.8 


84.2 


83.3 


• 77.4 


87.7 


• .23-. 32 


99.0 


92.9 


^ 94.5 , 


90.7 


90.01 


86.4, 


92.6 


.33-.42 


99.5 


97.8 


-97.3 


94.0 


%.l\ 


92.1 


97.5 


.43-. 52 


' 100.0*- 


> 98.9 


97.8 * 


96.2 


97 ; 8 


94.9- 


97.5 


.53-.62 




100.0 


. 98.9 ' 


'98.4 


98.3 


96.6 


98.1 


.63-72 






99.5 ; 


98.9 


99.4 


98.3 


99.4 


♦ .73^.82 






*99.5 


9,8.9 . 


100.0 


99.4 


100.0 


.83-.92 






100.0 , 


•98,9 




r 99.4 





* 1 

f « 

Note: All coefficients have reversed signs, so high indexes indicate 'good' targeting. The indexes in the middle 
set of rows in Sach subtable (- .22 tp + .22) have values 'that ar£* (approximately) not significantly different 
. from zero— or chance .targeting of services. 
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Table 7-20 / 

Cumulative Percentage Distributions by School and Grade: 
Product-Moment Coefficients Between Resource Cost of 
Instruction and Achievement Percentile Scores * 



Reading 



Index 


x.nooi 


UraOc 1 


uraoe * 


Grade J 


Grade 4 


Grade 5 


Grade 6 


Value 


(N m 2flfcV 

aw; 


(Kl — 1ft1\ 


in* ■ loJJ 




(N-180) 


(N- 178) 


(N- 161) 


Less than -.43 


0.5 


2.7 


2.7 


3.8 


0.6 


1.7 


1 Q 


-.42 - -.33 


0.5 


4.9 


3.3 


6.6 


1.1 


2 8 




-.32- -.23 


1.0 


7.7 


4.4 


7.7 




3.4 


5.6 


-.22- -.13 


1.9 


19.7 


6.6 


8.8 


7.2 


- 7.3 


8.7 


-.12 - -.03 


8.3 


34.4 


ll.5 


12.1 


11.7 


11.2 


13.7 


- .O2-.02 


in 7 






♦ icq 

i :>.y 


14.4 


11.8 


16.8 


.03-. 12 


24.3 


59.0 


22.4 


19.2 


22.8' 


17.4 


26.1 


.13-.22 


* 51.0 


74.3 


30.1 


* 29.7 


27.8 


25.8 


33.5 


.23.-32 


78.2 


86.9 


38.8 


37.9 


37.2 


36.5 


49.7 


.33-.42 


92.2 


94.5 


51.4 


52.2 


51.1 


59.0 


66.5 * 


*.43-.52 


98.5 


97.8 


65.6 


66.5 


67.2 


73.p 


' 79.5 


' .53-^62 


99,0 


99.5 


82.0 


* 81.3 


85.6 


83.1 


90.7 


.63-.72 


100.0 


100.0- 


93.4 


92.3 


96.1 


92.1 


95.0 


.73-.82 






98.9 


97.8 


* 99.4 


97.8 


98.8 


.83-.92 






100.0 


*98.4 


99.4 


99.4 


* 100.0 








Math 










Index 


School 


Grade 1 


Grade 2 


Grade 3 ' 


Grade 4 


Grade 5 


Grade 6 


Value 


(N-206) 


(N-183) 


(N« 183) 


(N-181) 


(N - 180) 


(N-177) 


(N-161) 



Hess than -143 


0.0 


1.6 


^2.3 




4*4 


1.7 


1.7 


1.9 


-.42 - - .3J 


0.5 


4.4 


8.7 




8.3 


7.2 


5.6 


5.6 


-.32 - - .23 


1.0 


9.3 


16.4 




13.8 


10.0 


' 11.3 


9.3 


.-.22- -.13 


7.3 


16.9 


21.3 




21.0 


17.2 


18.1 


14.3 


-.12 - -.03 


18.9 


31.7 


28.4 




27.1 


23 v 9 


29.4 


23.6 


-.02-.02 ^ 


29.6 


41.0 


35.5 




34.3 


29.4 


33.9 


27.0 


.03-.12 


50.0 


57.9 


48.6 




47.0 


43.9 * 


45.2 


41.6 


.13-.22 


73.8 


72.7 


63.9 




59.7 


60.6 


55.4 


55.3 


.23-.32 


94.7 ' 


83.1 


77.0 




7V.8 


78.3 


66.7 


' 72.7' 


.33-.42 


97.4 


92.3 


88.5 


84.5 


85.0 


76.8 


85.7 


- .43-.S2 


99.5 


, 96.7 


**94.5 




90.1 


92.8 


86.4 


94.4 


• .53-.62 


100.0 


99.5 


97.8 




94.5 


98.3 


94.9 


97.5 


.63-.72 




100.0 


98.9 




96.7 


100.0 


98.3 


100.0 


- 73-.82 1 






98.9 




98.3 




99.4 




.83^.92 






100.0 




98.3 




100.0 





Note- All coefficients have reversed signs, so high indexes indicate 'good' targeting. The indexes jn the middle, 
set of rows in each subtable (- .22 to + 22) have values that are (approximately) not significantly different 
from zero— or chance targeting of services. 
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example, if the students are ranked by need and the n most needy are not all in the sample 
of size n that is selected, then the method considers all possible selections of n students that 
would result in a set of^students less needy than those actually selected. 

( Although in concept the ^ie^hod deals with ranks' of* students, in fact, the procedure 
compares mean needines^ores of the two subsets, and a variance term is derived 
from the concept of all possible subsets. Then 3 Z statistic is computed and compared to a Z 
table to obtain, a value for alpha. For this procedure tgjiave a theoreticaf basis, it is 
necessary to assume that the difference between the/neans, considered as a variable over 
all possible ><ti>fons of the'N students into subsets of sizes n and N-n, is approximately 
normally distributed. 

Beaton's suggested index considers neediness as a variable of at least ordinal character, 
rather thten a dichotomous classification, and the ordering is specific to each school. The 
value of the index would be reduced if, for example, we decided to remove the most needy 
student Jrom the participating group and substitute the student who ranks tenth in 
neediness, even though both might be sufficiently needy to* qualify for CE. Thus, the mdex^ 
deals with finer distinctions than the dichotomous indexes discussed in this report, and 
more specifically addresses the within-school allocation of selection and services. 

There are also some serious disadvantages inherent in "this index, however. First, the 
difference between the means of the two subsets is not 'necessarily 'distributed normally; 
sortie tests of the method on artificial data sets revealed thaUhe Z statistic it generates is sen- 
sitive to a monotonic transformatitpn of thfe data, and to the shape of the distribution. These 
characteristics seem undesirable in light of the rational for the statistic, which is based 
entirely on a ranking of students-ranks being, of course, unaffected by a monotone 
transformation. A second disadvantage is that thejndex does ffpt have a conyer^nt and 
bounded range, as the previous indexes had. This* makes interpretation less simple, since 
the theoretical range extends mfinitely in both the positive and negative directions. It is not 
obvious whether a value of 2.0, for exampje, is large*or small. While it is possible to express, 
the unlikelihood of^ obtaining a given value of Z by chance, the statistic is not an easy-to-* 
read indicator of degree of association, r 

Table 7-21 contains the distribution of Beaton's index values;* by schoflf and by grade 
within* school, for reading and math. For reading" the school-level index 'values vary 
between *+ 1.8 and -0.4, with only one percent of the values being less than or equal to 
zero (Sincere mean of the CE group is subtracted from the mean of the non-CE group, the 
'good' values of the index are the positive ones,) The distribution is roughly bell-shaped, 
and is centered above zero (with a median of + 1.1). The ^chool-level distribution for math 
is very similar, but, shows somewhat poorer selection, with 91.3 percent of the values 
positive and a median value of +0.9. The indfc* agrees with the previous ones in that it 
shows somewhat worse targeting for math than for reading. 

A modified version of Beaton's index is also presented here, In keeping with the ranking 
rationale from which the statistic was developed, and in* response to its sensitivity to 
monotonic transformations of the data, the statistic was also computed on the ranks of the 

t * . « * . * 
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Table 7*21 



Cumulative Percentage Distributions by School and Grade: 
Beaton's Index 

■■ 4. 

Reading 



Index 


School 


i Grade 1 


Grade 2 


Grade 3 


Grade 4 


" Grade 5 


Grade 6 


Value 


(ti « 205) 


(N=143) 


(N = 169) 


(N=172) 


(N-164) 


(N = 156) 


(N= 144) 


Less than - 1 ,u 


0.0 


3.5- 


0.6 


0.6 


, 0.0 


0.0 


, 0.0 


-1:0 -'-0.8 , 


0.0 


4.9 


1.2 


U 


0.0 


0.0 


0.0 


-0.8 - -0.6 


0.0 


' 7.0 


i 1.2 , 


1.2 


0.0 


0.0 


~ 0.0 


-0.6 - -0-.4 


0.5 


9.8 


. 1.8 


1,7 


0.0 


. 0.6 


0,0 


n a n *> 
-U.4 - - U.z 


0.D 


12.6 • 


• 

2.4 


2.9 


0.6 


0.6 


2:1 


'-0.2 - -0.0 t 


1.0 


24 5 


3.6 


*4.1 


1 8 


1 3 


2.1 


0.0-0 1 


3 9 


35.7 


1 7 1 


5.2 


6 1 


2.6 


4.9 


jO.2-0.4 


1 1 .7 






7 n 
/.u 


i n a 
IU 4 


3.0 


8.3 


0.4-0 6 


19 0 


70.6 v 


17 2 


15.1 


17.7 


10.3 


* 11.8 


n (%.() ft 


OQ Pi ' 


P.7 Q 


07 O 


21 . j 


28.0 


17.9 


20 1 


0.8M.0 


50.7 


94.4 


37.3 


31.4 


42 1 


32.7 


# 36.8 


"1.0-1.2 


79.5 


97 2 


50.9 


50.6 


58.5 


44,2 


59.0 


1.2-1 4 


v 92. 1 % 


97.9 


72.8 


68.6 


80.5 


65 4 


< 75.7 ' 


1 A_ 1 A 


QQ C 




OQ O 

OO O 


84.9 


93.3 


80.1 , 


86.1 


1.6-1.8 


100.0 


99.3 


94:1 


93.0 


•100.0 


- 20.4 


9ff.5 


1.8-2 0 




100.0 


98.2 


, 95.9* 




96.2 


97.9 




• 




1 Math 










'index 


School 


Grade 1 


Grade 2 


Grade 3 


Grade 4 


Grade 5 


Grade 6 


Value 


(N= 161) 


(N*86) 


(N = %) 


(N=109) 


(N = 108) 


' (N = 108) 


(N=105) 


Less than — i u 


n n 
U.U • 


1 .2 


2.1 


3.7 


0.0 


0.0 


0.0 


-10- -0.8 


1.2 * 


• 47 


2.1 


3.7 


0.9 


0.0 


-1.0 . 


-0.8 - -0.6 


1.9 r 


9.3 


2 1 


5.5 


1 9 


W* 0.0 


1.9 


-0.6 - -0.4 


1 2.5 


12.8 


3 1 


9.2 


3.7 


* 0.0 


1.9 


n a no 
— U.4 - -U.z 


5.6 1 


1 7.4 


3.1 


10 1 


3.7 


0.0 


• • 

2,9 


no n n 
— u.z - - u.u 


Q 7 


23.3 


6.2 


1 1.0 


7.4 


0.9 


, 4.8 


0.0-0 2 ' 


11.2 




15-6 


1 3.8 . 


11 1 


6.5 


5.7 


0.2-0.4 


20.5 


43.0 


24.0 


M8.3 » 


19.4 


9.3 


16.2 


0:4-0.6 


30.4 


* 59.3 


3/3 


23.9 


30.6 


14.8 


31.4 


0.6-0 8 


47.2 


* 72.1 


45.8 


34.9 


43.5 


29.6 


: 42.9 


0.8-1.0 . ' 


67.1 


79.1 


61.5 


46.8 


57.4 


42.6 


56.2 


1.0-1.2 " 


eb.7 


88.4 


79.2 


67.9 


75.0 


60.2 


73.3 . 


1.2-1.4 


A\ 3 


97.7 ' 


89.6 


77.1 


83.3 


\ 76.9 


88.6 


1.4-1.6 


95.0 


98.8 


94.8 


84.4 


91.7 


90.7 


• 96.2 


1.6.1.8 


9a 1 


100.0 


99.0 


92.3 


97.2 


92.6 * 


98.1 


1.8-2.0 


98.8 




99.0 


96.3 


•■98 1 


96.3 


100.0 



115 143 



# needmess scores (instead of their means), within the school of withirvthe grade at a schobl, 
derived from the CTBS scores. The distributions are shown in Table 7-22 for reading and 
math. No appreciable differences 'in the index values are apparent. The median index value 
for reading is +1.1 and the median for math is +0.9, the same as that obtained for the 
unmodified version, discussed above. 

# 4 

Thus, for the data distributions with which we are dealing, the version based on percentile 
scores yields essentially the s^me results a* the modified version from ranks. Being simpler 
to compute, it is, therefor^, preferable to the method based on within-school ranks. Neither 
approach yielded median Z values that are significantly different from zero, but that likely 
reflects the statje of targeting more than the quality of the index. 

Having decided to examine rndexes based on ranks ofparticipants and non-participants, 
we find it natural to include-the Mann-Whitney U in the set of indexes to be considered. 
This statistic is a measure of the difference between the ranks of scores in two groups, and 
is, in fact, converted to a Z statistic for the finaf comparison. Like the other indexes already 
discussed in this section, therefore, it is a better measure of .difference than of association. 
We. can easily determine how unlikely any value of U would be, given the null hypothesis 
(and the assumed model), but it is not so straightforward to express the degree of agree-, 
ment between tHe dichotomous variable and the ranked variable in terms of U. The Mann- 
Whitney U technique is described in detail by Siegel (1956). 

T^ble 7-23 presents the distribution of the Z statistic derived from the Mann-Whitney U. 
(The Z values were selected for the fidex because their theoretical distribution is knpwn, 
thus giving us' some way to evaluate differences, and because they are comparable to the 
previous indexes in this section, which^are also expressablo^s Z Statistics.) The Z values 
< a present a more optimistic view of targeting than those derSed by the^previous method, 
because they have greater magnitude The meaning of 'the Z statistic is not quite the same 
under the two methods^ however: for the Mann-WKitney U it is an indication of the 
likelihood that the raqj<s olj^erved in the two groups could have occurred if there were no 
real differentiation between, the two groups, while the statistic suggested by Beaton deals 
with the likelihood that wocse rankings could have occurred. The median values from the 
Wann-Whitney U approach are highly statistically significant for both reading and math. 

* t * • 

Correlations with CTBS Ranks. The ranks of the CTBS reading and math scores can also 
be used to construct indexes through correlation, as the indexes of the first and second sec- 
tions were constructed.. The correlation coefficient constitutes an index with some useful 

, .characteristics, namely a bounded range and a center of zero. The use of the- ranks of the 
CTBS scores for a correlational index preserves the advantages of the statistic suggested by 
Beaton and the M^nn-Whitney U, as weft.. The ranks preserve more than a dichotomous 

\ distinction, treating need as more complex than mere presence or absence; they focus on 
t within-school differences among students, and they disregard the differences among scores 
tKat^ as argued previously in this section, are irrelevant to the concept of targeting, A cor- 

/ relation between the CTBS ranks anj^CE status (which is dichotomous) is a measure, then, 
of the agreement between the CE status and the ranks, indicating the degree to which the 
lowest achievers t^nd to be CE participants (and tend not to be non-participants). 

' * . \ • ' 
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Table 7-22 



Cumulative Percentage Distributions by School and Grade: 
Beaton's Index Modified by Ranks 



Reading 

/ ■ t 

Inde** 0 School Grade 1 Grade 2 Grade 3 Grade 4 Grades Grade 6 
Value-, (N-205) (N-143) (N-169) (N-172) (N = 164) . (N - .156) (N-144) 



Less than - 1 0 


0.0 . 


2.8 


1.2 


06 


0.0 


* 0.0 


0.0 


- 1.0- -0.8 


0.0 


4.2 


1.2 


0.6 *' 


0.0 


0.0 


0.0 


-0.8 4 -0.6 


0.0 


70 


1,2 


0.6 


• 0.0 


-0.0 


0.0 


-06- -0.4 


0.5 


10.5 


2.4 


1.7 ' 


0.0 


0.6 


0.0 


-0.4- -0 2 


0.5 


13.3 ~ 


3.0 


3.5 


0.6 


0.6 


1.4 


-0.2- -00. 


1.5 


24.5* 


36 


4.7 


1.8 


1.9 


3.5 


0.0-0 2 


4.9 ' . 


34.3 


7.1 


5.8 


6.1 


' 2.6 ~ 


6 2 



0.20.4 
0 4-0.6 

0 6-0.8 
0.8-1.0 
1.0-1.2. 

1 M.4 
1.4-1.6 



12.7. 


54.5 


10.7 


99 


10.4 


4.5 


9.7 


19.5 


69.2 


17.8 


16.3 


17.7 


If. 5 - 


n.8 


30.2 


85.3 


27.2 


21.5 


28.0 


19.2 


22.2 


50.7 


95.1 


36.1 


30.2 


42.1 


32.7 


38.2 


85.4 y 


97.9 


52.f 


55.2 


58,5 


46.2 


57.6 


98.0 ' 


97.9 


77.5 


74.4 


80.5 


74.4 - 


83.3 


100.0 

r 


99.3 


97.0 


95.3 i 


93.3 


. . 93.6 


95.1 



Index 
Value 



School 
(N-161) 



Grade 1 
(NU86) 



Math 

Grade 2 
(N»%) 



Grade 3 
(N-109) 



Grade 4 
(N m 108) 



Grade 5 
(N m 108) 



Grande 6 
(N « 105) 



Less than - 1 0 


0.0 


'3.5 


2.1 


2.8 s 


OS 


0.0 


0.0 


-1.0—0.8 ^ 


1.2 


4.7 


2.1 


5.5 


0.9 


0.0 


1.9 


-0.8 - -0.5 


1 2 


8.f 


3.1 > 


6.4 


2.8 


0.0 


1.9 


-0.6- -0.4 


2.5 


11.6 


3.1 


8.3 


3.7 . 


0.0 


1.9 


-0.4- -0.2 


6.8 


16.3 


3.1 


9.2 


4.6 


* 0.0 


3.6 


\^D - -0.0 


9.9 


26.7 




10.1 


8.3 


0.9 


4.8 


0.0-0.2 " 


13.7 


36.0 


15.6 


12.8 


12.0 


5.6 


8.6 


0.2-0.4 


21.7. 


44.2 


25.0 


19.3 - 


20.4 


* * 
8.3 


18.1 


0.4-0.6 


31 7 


58.1 


33.3 


25.7* 


30.6 


1,6.7 


32.4 


J0.6-0.8 


47.2 


72.1 


42.7- 


34.9 


44.4 


31.5 


43.8 


* 0.8-1.0 ' 


68.3 * 


80.2 


62.5 , 


51.4 


sU 


42.6 


57.1 


1.0-1.2 


* 86.3 


89.5 


81.2 


66*1 


73.1 


63.9 


76.2 


1.2-1.4 


94.4 


96.5 


91.7 


84.4 


90.7 


82.4 


90.5 


1.4-1.6 


98.1 
¥ 


100.0 


97.9 


95.4 


98.1 


..93.5 


96.2 
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Table 7-23 



Cumulative Percentage Distributions by School and Grade: 
Z Statistics Based on Mann-Whitjiey U 



■ Reading 



Index 


School 


Grade t 


Grade 2 


Grade 3 


Grade 4 


Grade 5 


Grade 6 


Value 


(N « 205) 


(N-143) 


(N=169) 


(N = 172) 


(N m 164)^ 


(N = 156) 


(N 1 144) 


* 

\0.0 


0.5 


8.4 


*0.6 


0.6 




1.3 


2.8 


W.J 






j . j 


D.L 


- ' • f\ i 

D. 1 


1 fx 


^ fx 
J.D 


10 . 


3.9 


51.7 


13.0 


11.6 


* 12.2 


7 1 


* 13.9 


1 5 ' 


6.3 

y 


72.0 


24 3 

* 


21 5 


-27.4 

* 


17.3 


23.6 


•2.0 


8.8 


* • 84.6 


33.1 


• 33J 


40.9 


30.8. 


32.6 


Z. j 


1 "X 7 
1 J / 


90 9 


44 4 


43.6 


53.7 


42.3 


no 

52.8 


' 30 


18.0 


93 7 


56.8, 


57 0 


e 65.9 


58.3 


64.6 


1 c 

J J 


24 4 


96 d 


66 9 * 


68 0 


79.9 


69.9 


74.} 


40 


30.7 


99 3 


h76 3 


^82.0 


; 85..4 


IbO.1 


86.1 ■ 


4 5 


37.6 


99 3 


82.8 


86.6 


93 3 


86 5 


87.5* 


50 


43 4 


100 0 


91 1 


92 4 . 


96.3 


90.4 


90.3 


5 5 


52 2 




94 1 


96 5 


*97.6 


93.6 

r 


92.4 


60 


61.5 




97.6 


98,3 


98.8 


96.8 


95.1 


65* 


65.9 




98.8 


98 8' 


98.8 i 




97.9 


7 0 


73.7 p 




98.8 


99.4 ' 


100.0 


. 98 7 - 


97.9 


75 


79.5 




100.0 


100.0 




1000 


98.6 


a 


■ r 




9 


Math* \ 








Index 


School 


Grade t • 


Cra*f* 


Grade 3 


Grade 4 


Grade 5 


. Grade 6 


Value > 


(N= 161) : 


%M = 86) 


-(N?,96) 


^(N=J09) 7 


*(N - 108) 


(N = 108) 


<N»105) „ 



0.0. 
0.5 
1'0 
1 5 

20. 
25 
30 
35 

4.0 
45 
5.0 
5 5 

6.0 
6.5 
7.0 
75 



^ I 9 
6.2 
118 
174 

23 6 
36.6 
42 2 
50'9 

58.4 
66.5 
73.3 
78.3 

84.5 
87.6 
91.3 
93.6 



*14.0 - 

:< 32 - 6 . 

41 9 \ 
59 3 i 

69.8 
84 9. 
90.7- 
^94 2 

96.5 
98.8 
98 3. 
100.0 



6.2 ^ 
W.7 , 
- 28J 
44.8 

59.4 
75.0 
82.3 
88.5 

90.6 
94 8 
97.9 
97.9 

100.0 



- 1.8 


^ 2.8 - 


• 5.5 


" 1 2.0 


^1 1 


is 20A 


36 7. 


?■% 


47.7-^r 


-58.3 


65 1 1 




78.9 3 


- * .81.5 


83.5 


? 92.6 






94.5 ' 


• \ 96.3 . 


94.5 


/96.3 


96.3 - 


99.1 


97.2 


100:0 1 


• 



0.9 
11 1 
•20.4 
32.4 



100.0 



£60.2 
74.1. 
81.5 

86.1 
92.6 
97.2 

"'! 

' 99.1 
1000 



1.9- 
12.4 
20.0 
40.0 

57.1 
77.1 
84.8 
8ff.6 

91.4 
94.3 
95.? 
98.1 

98.1 
99.0 
99.0 
400.0 



Note: Index values are midpoints of intervals of .5. 
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The cumulative percentage distributions for these correlations appear in Table 7-24 for 
reading and math. The reflected correlations are nearly all positive (at the school levei 98 
percent .for reading and 90.1 percent .for math), but do not tend to be strongly so. The 
med.an value is about +0.3*7 for reading and about .+ 0.24 for math. As with the previous 
..indexes, then;' there is evidence tf .better-tWchance targeting*, but the agreement 
between CE participation and within-schqbl achievement, rank is not impressive. , ' 

• The ranked .CTBS scores were also correlfc with the measures of . amount of services 
received. While these service measures dp not reflect CE/participation, or even, necessar- 
ily, services that are specifically part of CE, the Correlations of them with CTBS ranks 
indicate the degree of agreement between amount ofmstructional service-received and the 
need for CE services. Table 7-25 contains the-cumulativ^ percentage distributions for the 
reflected correlations, between the measure of need and the total exposure to instruction in 
reading and math. As with the previous indexes relating achievement to exposure the 
reflected correlations tend to be positive, but only mild'ly so. These analyses confirm tRe. 
earlier findings that exposure to services is not' a variable that contributes to favorable 
targeting indexes. - . « 

The corresponding distributions for the index based on t'pta'l resource cost for 'reading and 
math instruction appear in Table 7-26. The correlations for this variable show stronger* rela- 
tionships than those for'the'hours of services received:' Most 'of the reflected correlations' 
are positive (92.2 percent for reading and 82 percent for math), but the median valu&are- 
not particularly strong (about +0.22 for reading and +0.16 for math). The indexes based 
on resource^ost, then, are about as sensitive as those based on ^participation." ■ * . 

v DISCUSSION CfF ALTERNATIVE INDEXES • » 



Throughout this section we have taken the position that no one index {^targeting w/ll serve' 
for all situations against which targeting is* to be assessed-. What constitutes prope"r assign- ' 
rment of CE to students may be specified in various ways, depending on several considera- 
tions. Selection of a targeting, index for any given evaluation problem, then, requires that 
the evaluator exercise judgment; For this reason, we have with each of- the 25 indexes, 
presented the rationale' for its derivation and some empirical data on its distribution ,rn the 
••sarnple of SES schools. The purpose of this'chapter is-to summarize the discussion. of the . 
alternative indexes without the distractions caused by the concurrent development' of the 
indexes in the previous sections. . \ 

In order to discuss the wide range of indexes, .all 25 are named in the paragraphs/below: 
Eachjpdex is also provided with an acronym that can serve as a mnemonic. The acronyms 
wj]U>e employed in this chapter for the sake of brevity in the tables that follow. * 

The first indexes developed, - ; \ 

1. P.BELOW.50 CE participants scoring below the 50th percentile • 

2. P.BELOW-40 _ CE participants scoring below the, 40th percentile • 
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Table 7-24 . ^ 

Cumulative Percentage Distributions by .School and Grade: Reflected * « 

Point-Biserial Correlations Between Ci Participation and Achievement Rank * w 











Reddir 


8 




V 






inuc* 


j>\.noot > - 


ijrdoe i 






A A 

Lirade 4 , 




CrxtKs* A 




Value 


(N - 205) 


(N» M3) 


(N k - 169) 


*(N- 172) 


iN -.164)- 


(N« 156) 


(N-144^ 




- 20 or less 


i 00 


7.0 


1,2 


1.2 . 


^0.0- 


•0 6 


"iT 

. :-o.o 




.- 15 


'^0 0 


9.1 


2.4 


, 1.7 


0 0 


0.6 


' . 0.7 




- 10 


0,5 


1 1 2* 


~ 2.4 / 


2*3 


0.6 


0.6 


0.7 




» n5 

— Uj 


1 n 


1 ft*7 


1 A 
J.O 


4. 1 


1 i 


i .j 


1 A 

1 .4 




IJU 


7"H 
Z.U t ^ 


74 5 

Z4 J 
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20 
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/O.Z 


Z4 y 


zu.y 
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.30* 


43.4 


83.2 


4 29 6 


26.7 , 


34.1 


24.4 


42.4 




35, 


4 54 6 


-90 9 


•34.9 


33.7 * 


43.9 


32.7 


54.9 




" .40 * 


70.7 


95 1 . 


40 8 - ' 


41.3 


52.4 


44.9 


63.9 ' 


* 


45 


84 4 


* ■ Q7 7 

7/ .£ 


? 4Q 1 
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*^Q A 


70 ft 

/ u.o 




.50 
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98' 6 


58.0 
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-.55 
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7/ 


Qft A 
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* 










IIIUCA 




vjraoe w 










drdue o 




Value 


.(N*161) 


' (N » 86) 


(N^*96) 


(N » 109) * 


"(N » 108) 


(N * 108) 


(N » 105) 




> 

- 20 or less 


0.6 


8.9 


• "2.1 


5.5 J 


2.8 


0.0 * 


1.0. 




-.15 


1.9 


. * 10.5* 


3.1 


6.4 ' « 


•3.7-. 


• , 0.0 


i.o ; 




-•.10 


3.7. 


^ 14 0 


3.1 


9.2 


3.7 


0.0. 


. 1.9 




— .UJ 


A ft 
0 0 


1 7 2 


5. 1 


Q 7 

7.Z 


A 5 
0. J 


u.u 
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Q Q 

7.7 


* 7Q 1 
• Z7. 1 


7 1 
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ft 1 

-O.J * 
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4 ft 
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1 5 5 


1Q 5 

. J7.J 


1A 7 

to./ 


1 1 ft. 
i j. a 


■111* 


1 Q 
I .7 


1 7 A 
1 Z.4 




in 


74 ft 

£t O 


•4i n 

t J.U 


75 n 


1A 4 


. 1 1 Q 

* 1 J. 7 


17 0 


1 A ,7 
t 1 O/Z 




' * .15 


36.6 


• 48:8 


35.4 - 


22i. 


-29.6 


21.3 ' 


29.5 - 




.20 


45.3 


61.6 


43.7 "" • 


29.4 


37.0 


*»25.0 


40.0 




. j 


' A7 7 


77 It 


51 n 


44 n 


4A 1 
it *fO. I 


17 n 

J/ .u - 


5A 7 
jO.Z 




- \30 


78.9 J 


81.4 


• 54.2 


55.0 


56.5 


47.2 


66.7 




.35 


88.2 


' 89.5 


64.6 


64.2 


69A 


57.4 


75.2 




.40 


9h9 


, 89.5 


76.0 


71.6 


83.1 


* 69.4 


, 80.0 




♦ .45 ' 


95.7 * 


91.9 


. 84.4 


78.9 \ 


88.0 


• 75.9 


84.8 




.50 


, ^97.5 


' 97.7 


- 92.7 


% 83.5^ 


- 91.7 


82.4 . 


91.4 




.55 


98.8 - 


• 97.7 


'96.9 . 


89.0 


95.4 


88.9 


94.3 n 




.60 


98.8 


98.8 


96.9 


92.7 


97.2 


95.4 


96.2 n 




.65 


98.8 


* 98.8 


97.9 


94.5 


100.0 


99.1 


98.1 




.70 


100.0 


98.8 


97.9 


-95.4 




99.1 


99.0 . 




.75 




100.0 


99.0 


97.2 




99.1 


99.0 


* 


• , .80 <- 






99.0 


97.2 




100.0 


99.0 



Note.: Index values are midpoints of intervals of .05. 
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Table 7-25 



Cumulative Percentage Distributions by School and Grade: Reflected Product- 
Moment Correlations Between Total Exposure to Instruction and Achievement Rank 



Reading 



index 



School 



Value 


(N»2&) 


- .40 or less. 


> > 0.0 


- 35 


0.0 


-.30 


1.0 


-.25 


2.9 


-.20 <. 


6.3 


-.15 


9.7 


- 10 


14.6 


- 05 


26 7 


00 


43.2 


05 


58 3 


10 


72.8 


15 


83.5 


20 


87.9 


) ^ " 


93.2 


/ 30 , 


- * - 95.6— 


\. .35 


> 97,} . 


.40 


98 5 


.45 


99.5 


*.50 


'99.5 


.55 


99.5 


.60 


99.5 



Grade 1 
(N»I83) 



Grade 2 
(ISUI83) 



Grade 1 
(N - 184) 



Grade 4 
(N - 180) 



Grade 5 
(N-178) 



Grade 6 
(N»162) 



6.0 
?.7 
12.0 
18.6 
20.8 
27.3 
33.9 

37.7 
43.7 
49.7 
59 6 
64.5 
69.9 
72.1 

76.0 
81.4 
85.8 
89.1 
89.6 
90.7 
93.4 



2.7 
6.0 
8.7 
14.1 
15.8 
20.1 
25.5 

28.8 
34.8 
39.7 
49.5 
56.0 
63.6 
69.6 

75.0 
81.0 
83.2 
88.0 
91.3 
93.5 
95.7 



3 3 
4.4 
6.7 
8.9 
13.3, 
144 
23.3 

30.0 
36.1 
40.fr 
4^.9 
55.0 
59.4 
63.3 

68.9 
73:9 
77.2 
81.7 
87.8 
90.0 
93.3 



2.8 
5 J 
6.7 
7.9 
11.8 
15.7 
23.0 

30.9 
34.3 
39.1 
46.1 
52.2 
62 9 
68.5 

73.0 

80.3 

83.1 

87.6 ' 

89.3 

91.6 • 

93.8 



6.2 
7.4 
8.6 
11.7 
17.9 
22.8 
32.7 

36.4 
46.3 
51.2 
56.2 
648 
67.9 
71.6 

79.0 

81.5 

84.0 

86.4 

92.6- 

95.1 

96.3 



? Math * 

Index School Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 

Value ^(ISU206) ISUI83) (N = I83) (N = 183) (N = 180) (N=177) (N-162) 



- 40 or less 
-.35 
-.30 
-.25 
-.20 
15 

. -.10 

-.05 * 
.00 

05 
.10 
.15 

20 
.25 

.30 
.35 
.40 
.45 
.50 
.55 

.60 1 



1A 


5.5 


9.8 


7.7 


6.7 


1.0 


5.5 


13.7 


8.7 


10.6 


2 4 


10.4 


16.4 


11.5 


11.7 


5.8 


13.7 


20.2 


18.0 


-16.1 


9.7 


21 3 


-23.5 


24.0 


21.7 


18.0 


31.1 


32 2 ■ 


30.6 


30.6 


31.6 


404 


39.9 


39.3 


40.6 


45 1 


53.6 


47.0 


47.5 


50.0 


65.0 


62.3 


57.4 


54.6 


59.4 


77 2 


67 2 


63.9 


63.4 


66.1 


89.8 


76.0 


75.4 


68.9 


71.1 


92.2 


803 


82.0 * 


79.2 


77.2 


, 94.7 


87.4 


86.3 


83.1 


82.8 


97.6 


907 


91.8 v - 


86.9 


88.9 


99.0 


93.4 


92.9 


'90.2 • * 


89.4 


,99.0 


96 2 


96.7 


93.4 


92.2 


99.5 


97.8 


97.3 


94.0 V 


96.7 


99.5 


99.5 


97.3 1 


* 94.0 


98.3 


100.0 


99.5 


97.3 'h • 


95.6 


98.3 




100.0 


97.3 . 


97.3 , 


98.3 






98.4, * * 


98.4 


98.3 



5.1 
7.9 
12.4 
16.9 
26.0 
33.3 
41.8 

50.3 
~56.5 
63.8 
68.9 
73.4 
77.4 
83 1 

86.4 
89.3 
92.1 
93.8 
94.9 
94.9 

fe 4 



3.7 
6.8 
8.6 
17.3 
24.1 
37 0. 
48.8 

59.3 
N 67.3 
75,9 
80.2 
82.7 
88.9 
89.5 

92.6 
96.3 
97.5 
97.5 
97.5 
97.5 
98.1 



Note: Index values are midpoints of intervals of .05 
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Table 7-26 



Cumulative Percentage Distributions by School and Grade: Reflected Product-, : 
Mom^rCo'rXtions Between the Resource Costs of Instruction and Ach.eWnt Rank 



Reading 



Index 
Value 



School 
(N = 206) 



30 or less . 


1.0 


7.7 


-.25 


1,0 


7.7 


-.20 


1.0 


1 1 5 


-.15 


1.5 


19 7 


-.10 


3.9 


25.1 


-.05 


7.8 


34.4 


.00 


11./ 


42 1 


05 


17.0 


50 8 


. 10 


' 23 3 


59 0 


.15 


38.8 


66 1 


~20 


54 9 


74.3 


25 


68 4 


79.2 


30 s 


80.1 


' 86 9 


35 


-86 9 


91 3 


40 


93 2 


94.5 


45 


96 6 


96 7 


50 


98 5 


97 8 


55 • 


. 99D 


98 9 


60 * 


.99 0 


99.5 


65 v 


99 5 


100 0 


.70 


1000 





Grade 1 Grade 2 
(N = 183> .(N = 183) 



3.8 
4.9 
4.9 
6.6 
8.7 - 
11.5 
14.8 

19 1 
21 3 
23 5 
29 5 
35 0 
41 0 
45 4 

54.6 
62.8 
68.9 
77 0 
86.3 
91 8 
96.2 



Grade 3 
(N = 182) 



Grade 4 
(N = 180) 



Grade 5 
(N = 178)> 



7.1 


2.2" 


3.4 


7.7 - 


3 9 


3.4 


7 J 


6.1 


5.1 


8.8 


78 


8.4 


10.4 


8.9 


9.6 


11.5 


- 12.2 


11 2 


14.8 


13 9 


11.8 


15.9 v- 


172 


14.6 


18 1 


21 7 


18.0 


35 3. 


24.4 


21.3 


30.8 ^ 


26.1 


25.8 


36.3 


32.8 


33.7 


40.1 


38.9 


41.6 


47 3 


49.4 


50.0 


50 0 


55.0 


59.b 


59.3 


61 7 


67 4 


70 3 


76.1 


73 6 


78.6 


82.8 


79.8 


88. S 


88.3 ' 


86 5 


92.9 * 


93.9 


91.6 


95.6 


97 2 


"93.8 



Grade 6 
(N = 161) 



43, 

5 6" 
♦ 6.8 

8.7. 

9.9 
13.7 
^7.4* 

22.4 
26.7 
29.8 
36.6 
46.6 
51 6 
61.5 

67.7 
73 3 
78.9 
87.0 
91.3 
96.3 
96.9 



Math 



Index 
Value 



School 
(N = 206) 



Grade 1 
(N=183) 



Grade 2 
(N = 183) 



Grade 3 
(N = 181) 



30 or less 
' - .25 

- 20 

- 15 

- 10 
*- 05 

00 

. .05 
10 
15 

■ 20 # 
25 
• 30 
35 

40 
45 
50 
55 
60 
65 
70 



1.0 

1.5 

3 9 

6.8 
11 2 
180 
29.6 

42 7 , 
52 4 
65 5 
74.3 
87.4 
92.7 
96 6 

98.1 
99.5 
1000 



7.7 


12.0 


11.6 


9.8 


158 


14 9 


14 8 


18.6 


18.2 


17.5 


21 9 


' 21 5 


21 9' 


24.6 


24.9 


31 1 


27 9 


27.6 


41 5 


37.2 


33 1 


47. t) 


43.2 


39? 


57.9 


48 6 


45.9 


63 4 


60 1 


53.0 


73 2 


-65 6 


59.7 


76.0 


71.0 


67.4 


83.6 


76 or 


72 9 


89 1 


81.4 


79.6 


91.8 


87.4 


84.0 


94 5 1 


91 3 — 


. 87 8 


97.3 


V 94.0 


90.6 


98.4 


95.6 


■ 92.3 


- 99.5 


96.7 


95.6 


\ 99.5 


97.3 


%.] 


r 99.5 


98.4 


1 97.2 



Grade 4 

y 



grades 
(N-177) 



Grade 6 
(N-161) 



8.9 


7.3 


6.8 


10.6 


12.4 . 


8.9 


13,3 


16.9 , 


11.2 


i7.te 


18.1 


13.7 


20.0 


24.3 


16.8 


23.9 * 


28.2 


22.4 


28.9 , 


32.2 


273 


'389 


, 40.7 


34.8* 


45.(? 


44.1 


38.5 


'51.7 


50.3 


45.3 


62 8 


• 54-8 


53.4 


72.2 


63.8 


646 


77.8 


67.2 >• 


73.9 


82.2 


. 74.0 


83.2 


85.6 


78.5 ' 


87.0 


89.4 


84.2 


91.3 


92.8 


88. f 


93.8 


96.7 


93.2 


96.9 


99.4 


96.6 


< -98.1 


100.0 


98.9 


98.8 


99.4 


99:4 



Note Index values are midpoints of intervals of 05 
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3. . P.BELOW.35 ^ CE participants ^scpring belo\V the 35th percentile - 

could if computed on a cutoff acceptable to-the evaluator, minimally indicate the degree 
to which schools comply with the simple$^t$rpretation of Title I regulations. They are qq{ 
recommended 'for any other purpose, ^ however, because ot their insensitivity to the 
needinessof students who are not participating in.CE. • 

r r ,ir 

+ The unadjusted phi coefficients of the second Section, 

% 4. PHI.CES.50 Phi coefficient 'between CE participation and achievement 

scores dichbtomized^Uthe 50th percentile 

4 

5. PHI.CES.40 Phi coefficient, between 'CE participation and achievement 

scores, dichotomized at the*40th percentile 

. 6 . PHI.CES.35 Phi coefficient between CE participation and achievement 

• , scores dichotomized at the 35th percentile 

7. PHI.CES.TJ / Phi coefficient between CE* participation and teacher's 

judgment of need for CE . 

• are most appropriate for the assessment of targeting when the entire process of allocating 
CE to students, rather than just within-school C£ participation, is of interest; when need for 
CE is-seen as a^ dichotomous, nationally-normed variable, and, when it is participation 
rather than amount v>f CE service that is measured* • ■ - : -' 

' * V_ * : 

When the phi coefficient <s adjusted for unequal marginals, - * ^ 



^ ^ 8. CPHI CE.40 m Corrected phi coefficient betwfeen CE .participation and 

achievement scores dichotomized at the 40th percentile < 

9 CPHI.CE.35 'Corrected -phi coefficient between CE participation and 

achievement scores dichotomized at the 35th^percentile 



10 CPHLCETJ ^ - Corrected phi coefficient" between CE- participation and 
1j\ • teacher's judgment of need for CE ' ^ 

Jthe focus is more sharpiy on within-school-targeting, -but the other considerations still apply. 

When dichotomous indicators of need^are correlated with, level of services/ J 

IK R INSTR.40 Correlation between total hours of instruction received and 

achievement scores dichotomized at the 40th percentile 

12 R.INSTR.35 Correlation between total hours of instruction received and 

achievement scores dichotomized at the 35th percentile 
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13. R.INSTR.TJ 



14. R.RCOST.40 



15. R.RCOST.35 



Correlation between total hours of instruction received and 
teacher judgment^ need for CE 

Correlation between resource-costs of instruction and achieve- 
ment scores dichotomized at the»40th percentile 

Correlation between resource-costs of instruction and achieve*- 
ment scores dichotomized at the 35th percentile r 



16. R.RCOST.TJ Correlation between resource-costs of instruction and teach- 

er's judgment of need for CE 

the restftfmg indexes address somewhat different questions. Rather than the targeting of CE 
to needy students, these indexes measure the targeting of instructional services to the 
needy students. (It is important to note that these analyses do not identify CE instructional 
services specifically; indeed, it is not possible to differentiate accurately which services are 
associated with what programs.) These indexed would be most appropriate when the ques- 
tion is whether need^^ldents are receiving the most instructional services, disregarding 
any labels associatedNAa^» being selected for a CE program. The major disadvantage with 
these indexes arises from, the difficulty in measuring the instructional services and the 



resource-costs. 



The indexes from the third section are all correlatiipns between achievement percentile 
scores and either CE participation rates or levels of instructional service. 



17. R.CE.PERCN 



Correlation between CE participation and achievement per- 
centile scores , 



18. R.CE.INSTR Correlation between CE participation and total h<?urs of instruc- 

tion received - , <W 

19. R.CE,RCOST Correlation between CE participation and res6urce : costs of in- 

struction „ : > 

Like the indexes based op dichotomizations in the second section,_tHese examine associa- 
tion between neediness as defined by national norms, and CE participation rates or levels of 
service. They differ only insofar as the percentile scores represent a finer ordering, natipq- 
ally, than the cStichotomous versions, and so they are appropriate when the dichotomy- 
based indexes are not useful or desired. In practice, these pe;centile-based indexes would^ 
be easier to compute because the percentiles have to be obtained in eithe/ case, and usfrig^ 
thenr} directly saves the -effort of dichotomizing before computing the correlation coeffi- # 
cients. The major conceptual disadvantage ..of these indexes is that they indicate only the* 
strength of the linear relationship between CE participation rates or service levels ar 
achievemertF*percentiles. This seeyis to imply that a. five-point difference betvyperTtwo 
scores has the same practical significance (in terms of CE participation) at all pfoints along 
th# score range. 
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Finally, the indexes from section 4, 



20. IN D.BEATON 

21. MOD. BEATON 

22. y*CE.ARANK 

23. R.CE.ARANK 

24. R.INS.RANK 

25. R.CST.RANK . 



are based on within-school ranking of students by {heir achievement levek. These indexes 
are particularly expensive to compute for a large sample because they\entail racing 
studepts within-each school. For a school evaluation, however, the. use of local rank order- 
ing focuses oa targeting within the school in the most rigorous way, and the added expense 
to compote ranks for a single school is probably not of great consequence. 

Sensitivity of Indexes. In^additiori to knowing whicft indexes are conceptually' appro- 
'priate for a given application, the prospective user would benefit from some indication of 
the sensitivity, of^ the index. As 'each index was presented in the previous sections, a 
cumulative 'frequency distribution was provided. The distributions show the central 
tendencies of the indexes, their scatter, and the general shape of their dfstributions. It is 
tempting to consider ^he scatter of the distribution as an indication of sensitivity, since 
greatef?,variation does indicate that the schools are being differentiated more. Two caveats 
shouftJ hp kept in fnincf, however: first, it would be, misleading to compare the ranges $f 
indexesl^ftb known sampling distributions that differ, such* as the Z -statistic and most 
correlation coefficients; second, the differentiation of schools may be in large part caused 
by factors ^relevant to the concerns about within-school targeting (i.e., they may reflect to 
sorr^ degree the level of CE funding in a school, or the wa^ the school focuses its CE 
services at different gfades). ; , ' « 

It is'also appea\n£to think of the central tendency as an Jndicator of sensitivity. If we know 
a pndriAhat targeting is excellent in one set of schools attd is very poor in another set, 
then we 'would prefer the index that best differentiates between the sets. Similarly, if we 
knetw that all schools have excellent targeting, we would prefer the index for which 
targeting seems best. Even if we only knew that schools tend to have good targeting, the 
same decision rule would apply. Unfortunately, we have no such knowledge, and an index 
that shows schools in the -best light is not necessarily better .than one that presents a less 
^optjmj#tc picture. 



Index suggested by Albert Beaton 

Modification^ the index suggested by Beaton 

Mann-Whitney U between CE participation and achievement- 
score rank 

v • 

Correlation between CE participation and achievement-score 
rank 

Correlation between total hours o? instruction received and 
achievement-score rank «. , 1 \ 

Correlation between resource-costs of^ instruction and achieve- 
* ment-score rank / ' \ . 



On the basis of our work with the indexes? however, we feel that most schools are attempt- 
ing to honor the intentions of CE programs in allocating CE status, services, or costs to those 
students who- are defined as needing them. Misallocations do occur, either by error in 
determining need or by interpretations of guidelines and regulations of the CE program that 
militate against a perfect relationship between the need and the status, service, or cost. 
Such errors or misinterpretations, although numerous, do not appear to us to outnumber 
the instances of good targetrng. ^ * ^ 

Empirical Test of Validity of the Indexes. In order to obtain some basis for inferring the 
validity of thejndexes, we followed an approach similar to the methods used in validating 
some tests and test items. We first assumed that there was ^sensible basis for all, or most, of 
the indexes. Therefore, we could treat an index as valid degree that it correlated 

strongly with the other indexes. Jhe matrix of intercorrelafibns among all 25 indexes is 
presented in Table 7 : 27, the coefficients based on indexes for reading 'above the, diagonal 
and those based on indexes for math below the diagonal. 

^The correlation coefficients in Table 7-27 have surprisingly low values. Thj^s indicates that 
our indexes are not reflecting the same aspects of targeting. From inspection oMhe matrix 
of coefficients we can see that the'indexes for reading are interrelated quite similarly to the 
indexes tor math. Further, the clusters of high coefficients are also similar for the two 
subject areas. Four clusters appear to be consistent and strong, likely accounting fqc most of 
the^common variance in a factor-analytic sense. The clusters are: 

A * Coefficients based on phi t on Beaton's procedures, on the Mann-Whitney U, and on 
cbrrelations with achievement ranks. This is the largest .cluster of coefficients, and all 
the variables have the least incidence of negative correlations with the other 
variables in the matrix. It follows, then, that other considerations aside, a favored 
index should be selected from this cluster. The cluster includes indexes numbered 4, 
5, 6, 7,*17, 20, 21, 22, and 23. 

B Coefficients based on only the participants below certain cutoffs and thq$e based on 
corrected phi coefficients. This cluster appears to hav^ some* common base in the 
mere fact that they share the same numerical component. The cluster includes 
indexes numbered 1, 2, 3, 8, and 9. 
» * 

C Coefficients based on total Jiours of instruction are highly mterco^ated. The 
\^ cgmmon aspect, of course, is that hours of service i * the component. It mm follows, 
then, that indexes based on hours of service ate not particularly simiiar'to (or inter- 
changeable with) those based on CE participation. The cluster includes indexes 
numbered II, 12, 13, 18, and 24. ^ 

D Although moderately related to the cluster.based on hours of service, those based <3n 
resource-costs (a transformation of hours of service) tend to cohere into a cluster-by 
thetnselves We observeJhe phenomenon, then, that these indexes are not mter- 
^ changeable with those based on services or on CE participation. The cluster includes 
indexes numbered 14. 15, 16, 19, and 25. 
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Table 7-27 



Correlations Among 25 Targeting Indexes for Schools, Based on Over 200 Schools - 
(Coefficients above diagonal for reading; below diagonal for math. Decimal points omitted.) 



Indexes 
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89 97 
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Table 7*28 

Attributes of Each of 25 Targeting Indexes 



n. 

12 
13 
14. 
15. 



Index 



S.BELOW.50 
S. BELOW 40 
S.BELOW.35 
PHI.CES.50 
PHI CES.40 
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No 
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E. ^We expected a cluster of indexes based on teachers' judgments of neecf'for CE,i)ut 
such a cluster does not emerge with any strength. It seems that whatever unique 
information teachers' judgments being to targeting is submerged in other com- 
ponents of the indexes. 

- „ f; 

Recommendations. Finally, we wish to provide the reader with some Recommenda- 
tions regarding which f index migfif be, preferred. As the body of this report indicates, no 
recommendations can be made witRbut consideration of the circumstances in which the 
index is to be calculated or employed, therefore, by way of summary, we present Table 
7-28, which lists each of the indexes and indicates its attributes. Based on the entries of this 
table, the readerxx user should be able to rfrrive at an informed judgment about the relative 
merits of each index. 




* v CHAPTER 8. TEACHER JUDGMENT .OF NEED FOR 

COMPENSATORY EDUCATION 



Charles E. Kenoyer 

Teacher judgments of student need for compensatory education (CE) agree 
. moderately with scores on the CTBS. Jf a CTBS 'cutting score' at the 35th percen- 
tile is adopted, a near-maximum agreement rate (for.about three fourths of the 
students) is obtained. The accuracy of teach&Jbilgmentsiljiot influenced by the 
racial/ethnic or .economic characteristics of the students, the extent of .individual- 
' , ^ ization of instruction, Jiow students are selected for compensatory services, or 
teacher training and experience. 

r 

Teacher judgments one of the permissibly ways for schools to select students for CE, and is 
widely used (Wang, Hoepfner, Zagorsk', Hemenway, Brown, and Bear, 1978). It is impor- 
tartt, therefore, to know how well these judgments agree with objective measures of 
student performances. The Comprehensive Tests of Basic Skills (CTBS; reading and math 
subtests constitute the best measures of achievement that are available in* the Sustaining 
Effects Study, fSES), and so were used in these* analyses as the criteria against, which the 
judgments were gauged. The first step was to develop an index of agreement between 
teacher judgment of student needfor CE and the CTBS reading and math scores. 



SAMPLE SELECTION 

Teachers and students were selected for these analyses as follows. 200 teachers were ran- 
domly selected from 1 the SES teacher file at each grade. (Teachers associated with more 
than one grade w^te deleted.) A file linking teachers to their homeroom students was then 
jused to select all students fpr each teacher, because it was the homeroom teacher who 
provided the judgments regarding each student's need for CE. x - " > 

A reacher vvas subsequently, included in any analysis .except wRere one^or rnore of fhe 
variables were mjssmg. Many of the teachers taught no reading o7 namath, and so yvere 
excluded from 4 any analysis requiring such dafa.. Those who taught both subjects were 
included to the sample for each kind of analysis.* - * ? 

THE TEACHER JUDGMENTS " v 

* • * 

* ' O 0 4 V- 

The Student -Background Checklist iterp, in whi^h the teacher judges eacA student s need 
for both ceatfing and math'CE, was coded to yield to dichotomous classification decisions 
(whether the student needs reading CE or not, and similarly for math), which are treated 
independently (This approach is not based on an assumption that the two decisions are 
statistically independent of. each other, it is known from previous analyses that the 
judgments of student need for the two subjects ajre highly correlated. The present analyses, 
however, call for separate consideration of the two school subjects to synplify interpreta- 
tions (j)f the association between teacher judgment and test results for each.) The analyses 
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for reading -anc( math are based on nearly identical samples of teachers and students. 
Results for the two subjects are never analyzed jointly, but in parallel. 

The measu/e of agreement selected for the analyses is the correlation between a teacher's 
dichotomous classification of students by need for compensatory services in one of the sub- 
jects and the CTBS score for the same subject. Because it would be both inconvenient ancl 
costly to generate all the correlations that were needed— every correlation for each teacher 
in the sample-an approximate method was used. For each of the variables to* be cor- 
related, the sum and sum of squares were computed. If a missing value was encountered, 
the observation was deleted. For each pair of variables to be correlated, the sum of 
crossproducts was computed, .deleting any observation for which one of the .pair was 
missing. This procedgre leads to different subsampfes for the terms of the correlation 
formula, hence its departure from an exact correlation. It was assumed that each of the 
subsets was unbiased and representative of the whole sample, i.e., that the missing^)bser- 
vations w^ge not systematic. For each pair of variables x and y, then, the correlation was 
computed as: 

N N N 

xy , x ,N y 




i 

v 



The resulting correlations were transformed tb Fisher's Z for the subsequent steps in 
the analysis. * . . % 

We have no reason to expect that the validity of either teacher judgment of student need or 
ouf achievement measure is uniform over all grades, so the analyses were performed by 
grade throughout tfus report. Means of the correlations and their Z transfers (used for 
further calculahoTTTare presented in Table 8-1 for both reading and math. The transformed 
coefficients are called Teacher Judgment Index for Reading' (TJIR) and Teacher Judgment 
'Indgx for Math' (TJIM) and serve as indexes of teacher accuracy. Since teacher^udgments 
( indrcate need, and CTBS scores indicate achievement level, agreement is indicated by 
negative correlations That is, students who are judged to be .needy tend to have the lowest 
CTBS scores. ' f 

It is apparenKhat the agreement between teacher judgment and CTBS scores is poorest at 
the first grade, vfhere teachers had little opportunity to become familiar with the students 
or their backgrounds and where achievement t^st scores can be expected to be least valid. 
This holdsj rufifor both reading and math. Although there is some variability 'over the other 



grades, it is small in comparison to the differences x between .first and second grades. 

For many of the teachers in the sample, the corrections described above .Gould not be 
computed because the teacher judged all students the same, and therefore there *tas no 
variance for that teacher on the teacher-judgment maex. It was necessary jo-om"it those 
teachers from the (computation of the correlations reported in Table 8-1 . Ttenumbers and 
percentages ofjteachers omitted are shown by grade in T^le 8-2. 
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Table 8-1 



i 4- 

Teacher Judgment of Student Need. and Achievement Scores: 
Mean Correlations and Transformed Indexes, by Grade J 



1 

Grade* 


: z i 

Reading 




Math ' 




' Correlation 


TJIR 


Correlation' 


TJIm" 


i 
i 


- .13 


r — 

— 26 


-133 


r 

-.37 


2 ■ 


'- 60 


-.75' 


-.43 


-.50 


, ' 3 


- 55 - 


' -.68 


-.49' 


-.57 ' 


4 




\-.65 ' 


-.42 > 


-.49 , 

0 


5 


55_ 


\"-.68 


. \' -.49 




6 


' -54' 




' - 44 


; -.51 * 


Total 
Sample 


i" - 50' ? 




^ -.44 / 


* -.51 

«7 



■ . V ' .. ' ' ' • ' fable 8*2 .jri-. 

Number and Percentages of Teachers for VVhom Correlations 
' Could^dt Be Computed,, by Grade/ 



' Grade: 


•J 


'2** ' 3 * 


4 


5 


V 


Whole 
Sample 






. \ 


Reading 










Number • 


27 - 


19* , 


12 ' 


18 


16 


19 


111 . 


Percentage 




■ 11 


7 


10 


4t 9 - 


15' 


• 11 - 




«* 




Math 






**** 




Number 


37 


'36 


18 ' 


26 


25 1 


16 


158 


Percentage ~ 


22 


22 


10 


16 


16 


*2 


17 



This finding suggests that student-level correlations, by teachers, may underestimate the 
degree of agreement. Underestimation would result if a considerable number of the . 
teachers had only students who correctly fall into one teacher-judgment category, as would 
tend to occur, for example, in cases of ability grouping. Such clustering of high- or low- 
ability students would eliminate from consideration large numbers of students whose 
achievement 'Stores are consistent with their teachers' judgments. These students are not 
eliminated when correlations are computed over the entire sample, rather than by teacher 
(but stijl subset by grade). For this analysis, teachers are disregarded, and only one correla- 
tion is computed for reading, and one for math, at each grade. The correlations appear in 
Table 8-3. 
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Table 8-3 

Correlations Between Teacher Judgment and CTBS 
Scores at the Teacher Level, by Grade 



* Grade: 


-A 


1 


2 


3 


4 * 
> 


^ 5 * 


6 


Reading 




-.22 


-.60 


'- 62 


-.57 


-.59 


-.56 


Math 




-.31 


-.42 


-.51 


-.45 ; 


-.49 " 


-A3 



.Some of the correlations at the,higher level of aggregation (that is, not 'it the teacher level) 
are slightly larger than those computed at the teacher level, but the majority are somewhat 
smaller. For reading, the pooled (not-by-teacher) correlations are consistently larger, with 
only one exception at the first grade, but again the differences are small. This result suggests 
that the within-teacher correlations are, on the average, adequate representations oithe 
association between teagher judgment and CTBSiStores, and so can serve as a valid inoex 
of agreement.'* . f 



PERFORMANCE CURVES FOR THE TWO TEACHER JUDGMENT CATEGORIES r 

Given that the agreement between teacher judgments and CTBS scores is [ess than perfect, 
it is of interest to know whether there is spme level of performance on the QTBS that 
corresponds to- the boundary at which teachers make different judgments, and how well 
tea<cher judgments correspond to a division made at such a level. In order to examine these 
questions, students at each grade^lere divided into two groups according Jo whether they, 
were judged by theirleachers as needing CE in reading or math, or as not needing it. The 
cumulative percentage gf students in each group was plotted against percentile scores 
, for each grade and subject, as shown in Figures 8-1 and 8-2. (Every fifth percentile point 
is plotted!) ' N ^ 

It is apparent that jGdgment groups for both reading and math have less separation for first 
. gcade than for the other grades, confirming the relative sizes of correlations for the grades 
described previously. If first grade is disregarded, it is apparent that the two^udjjjment- 
'based groups are widely separated in the plot at about the 35th achievement percentile. 
For example, the classification of second-grade math students ofi the basis of Scoring above 
or below the 35th percentile would be con^stent with the teacher-judgment classification 
for 80 pqrc^nt of those judged not needy and about 70 percent of those judged needy. For 
many of the other subject-grade combinations, the agreement would be higher. In fact,, all 
the reading curves are more widely separated than the math curves/ 

This finding, indicates that using a point near the 35th percentile as a cutting point would 
optimize agreement between teacher judgment and CTBS classifications, and that over 75 
percent of the students would be classified the same by the two criteria. 
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Figure 8-1 

Cumulative Percentages of Reading Achievement for Students 
Judged To Need and Not To Need Reading CE 
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INFLUENCE OF OTHER STUDENT CHARACTERISTICS 

FoJr student characteristics, all reported by the same teacher who judged need, were 
selected as potentially influencing teachers' judgements, student ethnicity, (majority vs. 
minority), education levels of the student's parents (postsecondary education vs. no 
postsecondary education), student receipt or non-receipt of free or reduced-price lunch, 
and primarryjanguage spoken in the student's home. The impact of each of these variables 
was examined by employing it as a control variable in a partial correlation. (Actually, the 
partial correlation cannot be interpreted as an indicator of influence of the control variables 
on teacher judgment alone, since it includes their influence on both variables correla- 
ted, *but it serves to explore tfie Impact of the control variables on the correlation, as an 
initiafstep.) " t . 

/ 

For this analysis, both the average of correlations computed for each teacher and the 
correlations computed for the whole ^ample, by grade, were exammgd. The results were 
similar for both. Since the correlations at the whole-sample level were computed fn the 
usual -way while the ^acher-level correlations were approximated, the whole-sample 
correlations were selected to report here. They appear in Table §-4. v 

.In every case, the partial correlation is somewhat smaller in magnitude than the s&mple 
correlation, indicating some influence of the control variables on the relationship between 
teacher judgment and achievement. The influence is uniformly small, however, the largest - 
(for math in the third grade) being only .07T"The simple correlations themselves account for 
no mo(^ tban about 38 percent of the variance. The most accurate surrynary seems to be 

fable 8-4 

t ^ Simple and Partial Correlations Between Teacher*Judgment and * 
Athievement at the Whole- Sample Level, by Grade 
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that teacher judgment-tends to agree more thdn to disagree with achievement level, but 
neither variabfe is a potent predictor of the other (but compare the predictive power if a 
'cutting score' is used to make the CTBS score a dichotomy). When the student variables 
are partiated out, the correlation is reduced slightly, indicating thy^t student character- 
istics do not play a large role in the relationship betwe^ measured need for CE and 
judged need. ; 

INFLUENCE OF TEACHER/SCHOOL CHARACTERISTICS 

Several teacher or school characteristics may fnffuence a teacherVjudgensients of student 
need for CE The first of these to be considered is the basis of student alignment 1o CE 
progr^mi This information was obtained from an item of the Teacher Questional re, Part B 
(for reading), and an item of PartC (for math). Both the it^ms permitted multiple responses, 
and so could yield any combination of the following options: 

x • An achievement test score * ^j; 

• Teacher judgment 

4 • Parent request' ~ w 

• Student request 4 ) . ^ 

• •"'Other " } 

/ 

For the present analysis, the responses were coded to yield three response categories as 
follows if 'teaicher judgment' was marked; the response was assigned to the 'teacher judg- 
ment' category, regardless of other responses; if only 'an achievement test' score' was 
marked, it was assigned to the 'adffievement test' category, and if any oth'er combination of 
maYks vvas found, if was assigned fo the 'other' category. This last categofy, then, includes 
^ny multiple response that does not have 'teacher judgment' as a component. The same 
coding was performed on both the reading and math items. - . % ' . 

The three categories formed in thijs way, together with the si*grade levels, de/rned the celfs 
for a three-b^six analysis qf variance. The basis of student assignment *to reading CE was 
used in defining the-cells for Jhe,analysis in which the Teacher Judgment Index for Reading 
(TJIR) was the dependent variabfe, and^the corresponding item for math CE wa^used to* 
define the cells when the Teacher 'judgment Incjek for Math if JIM) was the dependent* 
variable Results of the analyses appear in Table 8-5 for TJIR and Table 8-6 for TJIMr 

^Confirming the previous pattern in the correlation means, the analysis of variance of the 
coefficients shows tftat JJIR aqd T}1M differ for different grade levels. This difference is to be 
expected on the basis of greater familiarity with the.students, and is .also influenced by the 
. increase in validity of the CTBS after the first grade". But the primary variable for which this 
analysis wa* performed is the Basis of Assignment, it seemed reasonable that the usual 
practice of assigmng students to CE could affecHhe teachers' perceptions of the importance' 
of'such judgments or rtieir levels of experience in making such judgments. This school 



Table 8-5 
Analysis of Variance for TJIR 



Source of Variation 


Sifln of Squares 


df 


Mean Square 


F 


Significance 


Grade (C) 


' J 22 795 


5 


4 559 * 


32.685 


0 000 


Ba^i7oTAssIgnment (6) 


0 153 


2 


0 077 


0 549 


0 578 


G X B Interaction 


1 5^2 


10 


0.158 


1 134 


0.333 


Residual 


144 782 


880 


0.165 










Table 8-6 






* • 




Analysis of Variance for TJIM 






Source of Variation 


Sum of Squares 


df 


Mean Square 


F 


Significance 


— 

Grade (G) 


. 3 809 


5 


0.762 


7.416 


0.000 


Basis of Assignment (B) 


0 052 


2 


0.026 


0.253 


0.777 


G X B Interaction . 


1 225 


"10 


0.123 


I 193 


0.292 


Residual 


80.024 


779 


0.103 







. characteristic has no significant effect on either TJIR or TJIM, either alone or in interaction 
with grade level. It appears, therefore, that a policy of making-assignments to CE on the 
basis of teachejjudgment has litjle or ho effect* on the quality of those judgments (as 
measured *by their correlations \vith achievement scores). « 

Three composite measures of teacher characteristics or school practice's appeared to he 
associated with accuracy of judgment, as^ indexed by TJIR and/or TjlM. One of these, 
Teacher Experience a*d Training, could equally"logically be related to either index, and so 
correlations were computed for both. The other two composites are indices of the degree 
to which the instructional approach is individualized, and are expressed separately for 
reading and math, so only the correlations between the reading individualization index 
and TJIR, and between the math individualization index and TJIM, were romputed. 
Correlations are presented separately by grade, in Table 8-7. 

Ail-but three df the correlations are lessJhan .1,, indicating thaUhe variable being correlated 
with TJIR or TJIM wbuld enable us to predict less than one hundredth of the variability in" 
th^index. The largest correlation, ,134, accounts for only *M8 oMhe variance. It is 
apparent, then, that none of these variables has any appreciable effetHipoo-the accuracy, 
of teacher judgment of a student's need for CE. 
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Table 8-7 



Correction of Teacher judgment Index With Composites 



Composite Measure 






Grade 
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-.05 


-.01 . 
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01 
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- v 08 
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PART llh MEASURING STUDENT GROWTH' 



Part III contains four chapters on the selection and development of the measures of student 
growth and two chapters that investigate issues of testing that have been raised. In the first 
chapter we describe how the Comprehensive Test of Basic Skills was selected as the 
standardized achievement measures for the study, and how the tests were debiasejd'.m an 
effort to make the scores apply equally well to all kinds of children. We describe the alter- 
native to this kind of testing in the second chapter. The criterion-referenced approach was 
carefully studied to determine its applicability to the SES. In the third and fouctkchagers, 
the selection and development of measures of functional literacy ahd computation and of 
affect are described in detail. Exhaustive searches for usabte instruments were mid** first. In 
the case of the measure of practical achievement, no test was found that could meet the 
needs of the study, so a new test was developed, field-tested, and revised. In the area^of 
student affect, a published instrument for measuring attitudes to reading and math was 
- selected, but the instrument was augmented with new it^ms that would bring the scores 
into closer agreement with the goals of the study. 

jWthe •fifth chapter we fook at 'out-of-level' testing— a practice whereby very low-aehieving 
^tfrdents are tested with a level of test designed for Students in a lower grade. 'Out-df-level 
testing has much to recommend it on several grounds, but it also presents problems. The " 
pros and cons are discussed in some detail. 

In the last chapter vye investigate the issue of the 'speededness' of the achievement tests 
used in the, SES. Oufspecial concern is fo determine if the speed factor influences scores 
differently for different racial/ethnic groups. In the event that such effects can be uncovered 
and speed is considered an irrelevant component of achievement, testing procedures in the 
SES should be modified to eliminate speed so that the findings are not influenced by it 
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CHAPTER 9. MEASURES OF ACADEMIC GROWTH FOR THE SUSTAINING 
. \ EFFECTS STUDY 



Ralph Hoepfner and Franqois Christen* 

After careful ^Juation of all available standardized achievement tests, the 
Comprehensive T^of Basic Skills (CTBSl Form S, was seleaed aJ the measure 
of growth in readingSmd math to be administered to all grades in both the fall 
and spring. The items di.the CT$S were studied for sociocultural bias and those 
identified as biased were\arkect\for elimJnatidn from totalgeores. Studies were 
also carried out to determinbuf alternate forms of the test should be used (they 
should not), if test levels 'shoul d 6e coun terbalanced in administration order 
during the^first year (the ordei^houid be^easier level first), and what kinds of 
score A should be prepared for later analyses (only reading and math total scores, 
. should be analyzed). The result^ of these efforts were a set of recommendations t 
, that would guide the achievement analyses for the remainder of the study. 

* / * 

Achievement growth in the areasw reading and math are afmost universally considered to 
be the^post important outcomes of pur educational system. These skills, along with the 
affective arid social skills that ari receiving increased attention by educators, enable the in- 
dividual to fuhction effedively/n today's complex, society and, to realize personal potential, 
FoMhese reasons, jhe measyfes of academic growth for the Sustaining Effects Study (SES) 
are of critical importance. 



SELECTING /HE MEASURE OF ACADEMIC ACHIEVEMENT 




There can be no doubr that published standardized achievement tests are the most widely 
'used and the most exhaustively studied measures of academic growth. In addition, their 
WoricahpopUlaritv/and acceptance have added to their perceived value as indicators of 
achievement gpowlfi. The task of selecting an achievenment test for a large national study 
of compensatory Education (CE) cannot be taken lightly. The use of standardized achieve- 
ment tests in ^jflucational evaluations has been criticized from many quarters, Carver 
(1974), for instance, found standardized achievement tests. to be too psychometric as 
opposed to edl/metric, claiming -that they focus too much on stable between-individual 
differences ratter- than on within-individual growth. Others, such as Horst, Tallmadge, and 
Wood (1974Vand Barker and Pelavin (1975), question the technical adequacies of achieve- 
ment test ay they are used for program evaluation.* > ~ 

The concern about, using stagdardized achievement tests in educational research has also 
been mounted as a result of research findings on the impaqt.of education on disadvantaged 
chilclrenff Most recent studies have shown that^educattonal programs such as Head Start, 
produce only. small measured effects on educational achievement. As a' consequence, 
resea/thers such as Campbell and Erlebacher (1970) have carefully scrutinized the 



hristen was affiliated with RMC Research^ Corporation, which was under subcontract to SDC to assist in 
electing the achievement tests j 
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methodology used in past evaluation efforts. Had research found that education produced 
iarge measured effects on disadvantaged children, there probably would be less concern 
about the methodology used in the evaluation of educational programs and less concern 
over* the adequacy of standardized achievement tests. The very refined statistical models 
now in vogue in, educational research make increasingly greater demands on the 
psychometric properties of achievement tests. 



Our review of existing achievement tests revealed that none had all the characteristics we 
desired for use in the SES Ojur strategy was, therefore, to select the instrument that best met 
/ our»needs from those available. 



The task of selecting the best instrument called'for a trade-off analysis of standardized 
achievement batteries on the basis of the objectives of the study. An advisory panel on 
nationally known testing experts was to assist us in^he development of selection criteria. 
Criteria were to be based upon a review of the study design, standards developed by the 
y American Psychological Association, reviews conducted by the Center for, the Study ■ of 
"Evaluation, and reviews sponsored by the federal go\ernmec(t in conjunction with other 

recent evaluation studies. The selected test was to: r ' 

» , " * 

• ^ ave empirically based fall and spring percentile norms 

• Span grades } through 6 

• Have reading (comprehension. and vocabulary) and math (concepts and computation) 
subtest for most grade levels * f ; 

• Assess basic skills rather that reflect a specific curriculum 

• Have levels convertible to a common metric that cuts across grade'levels 

• Have minium ethnic-group bias and be relevant, interesting, and meaningful to all 
students 

• Have representation of minority groups in the standardization sample 

• Have acceptable reliability and validity 

• Have parallel fdrms • ' . - 

• Be easily administered, scored/ and .processed* 

A screening procedure was applied to the CSE Elementary School Test Evaluations (Hoepf- 
ner et al., 1976), a large compendium of tests evaluated by the, Center for the Study of 
Evaluation (CSE) First, reading jests were screened using criteria developed by CSE for 
subject" appropriateness, norm appropriateness, ease of scoring, and scdfe mterpretabihty. 
The surviving tests were then reviewed in terms of grade-level appropriateness. A parallel * 
procedure w.<Js applied to math tests. % 
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Reading Test Selection i 

0 

Subject -Matter Appropriateness. The measurement requirements were first translated 
into content areas according to the CSE test evaluations. The following CSE educational 
goals were thought to be important in assessing reading proficiency: 

Coal 1. Word Attack Skills. Identifies or combines the sound components (not the 
meaning) of words using phonetic skills or structural clues. Knows co/respondence 
between sounds and their written representations. Identifies letters, syllables, rOQts 
prefixes, and suffixes. * 

Goal 2 Recognition of Word Meanings. Shows understanding of the meanings of 
written words, by identifing definitions, similar words, illustrations, synonyms, or, 
antonyms. Knows different meanings that the same word may communicate. 

Coal 3: Reading Comprehension. Understands material read. Infers the meaning of 
Jwords from context Follows written directions. Identifies topic sentences, mam ideas, 

and intentions of the author, and finds supporting details and illustrations in the texk^ 

Keeps track of temporal sequences, spatial order, and other relationships. Reads at a 

rate appropriate to the material and purpose. 

i * 
Coal 1, above, is appropriate only for grades 1 and 2, and serves as a general 'reading 
readiness' category. Most tests in this goal category are word recognition (without meaning) 
or syllable-tetter attack skills. Coals 2 and 3 are appropriate for grades 1 through 6 and even 
beyond Most tests io goal category 2 ask for the examinee to match words with something 
on the basis. of the read meanings of the word. Tests in the third goal are of the 'read a 
paragraph and answer the question' tyfie. * 

The first stage in the selection process consisted of eliminating those tests that do not have 
subscales assessing at least two of the above-mentioned goal categories (goals 1; 2, or 3 for 
grades 1 and 2; goals 2 and 3 for grades 3 through 6). On this basis/ 58 reading tests were 
'eliminated from further consideration. \ 

Norm Appropriateness. The next criterion used for screening was norm appropriate- 
ness At this stage we again used the CSE evaluation, although the CSE criteria are not as 
hgoYous as we desired. The CSE criterion for norm appropriateness is as follows: 

- / 

Is the norm group representative of the national population? Norms should be 

based on a sample of examinees that is drawn from a variety of family and com- 

munity backgrounds' and includes students at the grade level for which the testis 
being evaluated. Raters looked for explicit or implicit evidence that "the sampling 
was intended to be random or stratified^ recent, geographically balanced; racially 
and ethnically representative, and heterogeneous with respect to. population 
density ^he test or subscale was credited with 1 point if an attempt was clearly 
made to obtain a nationally representative and current sample. It was given no 
pointsNf there was no norm sample> or the sample appeared to be local or inci- 
dental (Hoepfner et al., 1976). 
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Forty-three of the remaining teats did ngi recene^a rating- of f on the ^SE' evaluation and 
wete consequently eliminated. • * . 

Ease of Storing The reading tests were ihen screened for ease of scoring. In a large 
study it is imperative to have a machiAe ; sct>r^blejest The CSE test evaluations do not cover 
the issu^ of machine scorability, but do rat? tests on ease of scoring. The CSE criterion is 
as follows. • * * - . , : \ 

* * * 
How easy and objective is the scoring procedure' Hand or machine scoring that 
.can be quickly carried put is, bf-<ourse,^eferable to more difficult and time- 
consuming procedures. $bjfin£ was judged to be objective and simple' When- 

"ever it consisted of some Straight-forward process. Tests fulfilled tfiis condition in 
many ways, providing answer sheets % , matching Pencils* templates, machine- 

, scoring sen/ices, or just uncomplicated scoring guides 

The test or sub-scale was credited with 2 points *f the scoring procedure was objective and 
simple Only one of the remaining tesis djd not earn 2 points, it was eliminated. 

♦ 

Score Interpretabihty The next, cuterion on which tests were screened was score ihter- 
• pretabilitv Here again, we us^d.the CSE test evaluations to eliminate test with unusual or 
naive score conversions 'Yhe ra^ore technical issues relating to gr^de-equivalent scores and 
to a score metric common q*er all grades, howexer, are discussed in greater detail later. 
The^CSE criterion was as follows: % 

» <- » 

Can the scores be easify interpreted relative to some norm group or standard? 
ConvertefcTscores that ate common, simple and clearly explained are desirable in 
order that jttPbol personnel will understand the scores and not* be misled by 
them Pass-lail; percentile ranks, stamnes, and grade equivalents are considered 
common and simple. The tester subscale wa$ credited with 1 point if common, 
simply interpreted conversions of test scores were provided] and 0 points if no 
conversion from raw scores is provided, or if conversions are novel, less com- 
mon, or unclear in their meaning. J 

One of the remaining tests did not meet 'this cnterion'and was eliminated. 

Crade-Leiel Appropriateness. The next screen was employed to eliminate tests that did 

not cover grades 1 .through 6. Six tests were eliminated. ' • % . 

• ** ^ * 

t Sunning Reading/Tests. The following, eight tests met the brpad and permissive screen- 
. ing criteria for the* measurement of reaoKng goals: , , * 

California Achievement Test )OWA Tests of Basic- Skills 

. Comprehensive Test of Basic Skills 4 * * Metropolitan Achievement Test 

Durrell Listening-Reading Series • SRA Assessment purvey 

Cates-MacGmitie ReadtngTe^t. ' "Stanford Achievement. Test 
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Mafh Test Selection 



The procedure used to screen math tests was parallehp the one adopted for reading tests. 
However, since'there are fewer specialized math testsyand since the CSE rules* for classify- 
ing a test in one goal category as opposed to another are perhaps more arbitrary for math 
than for reading, tests were not eliminated as to jgoal appropriateness on the first rourld 
unless they covered none of the goal categones^gutlmed below. 

Subject Matter Appropriateness. The following CSE educational goals were thought ^to be 
important in assessing proficiency in math: ^ 

Goal 1 Knowledge K)f lumbers and Sets. Understands numbers and fractions. Dif- 
ferentiates between numerals and between prime and composite numbers. Identifies 
factors, multiples, apd relative primes of a given number. Understands set member- 
ship, set relations, set correspondence, anc} operations with sets. Relates set notation 
arid diagrams to categorical statements in English. . . 

Coal J Knowledge of \umeral Systems and \umber Principles. Reads, recites, and 
writes numerals. Understands place values, the rounding of numbers, the decimal 
system of numeration, numeration with bases other than 10, and Roman numerals. 
Understands the commutative, associative, and distributive properties, inverse opera- 
tions, properties , of 0 and 1, negatives, and reciprocals. Understands number-line 
diagrams Finds and evaluates simple numerical hjles based on observation. 

Coal 3 Knowledge Basic to Algebra. Understands number relationships, ^number sen- 
tences, variables, and formulas. Reads sentences tising letters or frames and equality or 
inequality sighs, and relates them to quantitative statements in English. Solves or graphs 
equations and inequalities. Tests relations for reflexivity, symmetry, and transit+vity. 

Coal 4 Whole- \'umbe£ Computation. Adds, subtracts, multiplies, and divides inte- 
gers, checks answers. * - v 

Coal 5 Decimal and Percentage Computation. Adds, subtracts, multiples, and di- 
vides decimals, or decimals and integers, checks answers. Transforms-frac&ons into 
decimals, decimals into perce'nts, percents^into fractions, and vice versa. Reads and 
writes decimals and percents. Solves percentage' computation problems. " - 

Stxtv-nine tests with sub-scales that assess at least one category from goals 1 through 3 
(concepts) or from goals "4 through 6 (computation) were included for further screening.. 
Tests that fall into goal categories 1, 2, or 3 tend to have items in other goals also, although 
CSE'b procedure arbitrarily categorizes them into one goal rather than another. Tests that 
cover goals \, 2, or 3 ace^appropriate for grades 1 through 6 and beyond. Tests of goal 4 are 
appropriate for all grades, tests of goal 5 for grades 4 arid above, and tests of goal 6 for 
grades 6 arid above % 

Sorm \ppropr\ateneb6 Math testb that do not have appropriate norms according to the CSE 



test evaluations were eliminated. The CSE criterion is the same as was previously defined 
for the reading goals. Fifty-four tests did not meet it.. 

Ease of Scoring. The next screen of math tests eliminated those that did not meet the CSE 
test evaluation's criterion of ease-of-scormg, as previously defined. Only two tests were 
eliminated because they did not meet this criterion. 

/ 

Score Interpretability. The remaining tests were>ubjected to CSE's criterion of score inter- 
pretability as previously defined. Only one test did not meet the criterion. 

Crade-Le\el Appropriateness. Six, math tests did rpt cover grades one through six and were 
therefore eliminated. 

Sur\mng Math Tests. The following tests successfully fulfilled t^e general criteria for math 
tests for use -in the SES: 

California Achievement Test 
Comprehensive Tests of Basic Skills 
IOWA Tests of Basic Skills 
. Metropolitan Achievement Test 
SRA Assessment Survey 
Stanford Achievement Test 

Because the SES is focused on educationally disadvantaged children, it was expected that 
there would be a substantial amount of out-of-level testing— that is, children'would be 
given a test lower than that. recommended by the test publisher for a given grade (see 
Hoepfner and Wang, this volume). Out-of-level testing is used to ensur^that the test given 
is not so difficult for the students thai a meaningful measure of their.academic achievement 
cannot be obtained. Consequently, it isjjecessary to choose a test of academic achieve- 
ment that covers one grade level below the lowest grade level being tested. Since the 
lowest grade level tested in the SES is grade 1, the academic achievement test selected 
bhould cover kindergarten. Thus, for both reading and math, four of the surviving tests 
were eliminated because they did not cover kindergarten. The remaining tests under 
consideration were: 

* Comprehensive Tests of Basic Skills 
Cates-MacCmitie'Reading Tests 
Metropolitan Achievement Test 
Stanford Achievement Test 



Additional Selection Criteria and Visits to Test Publishers 

The expert panel on achievement tests made several recommendations for test selection. 
The panel first recommended that of the four test finalists two. be seriously considered and 
scrutinized The Comprehensive Tests of Basic Skills (CTBSj and the Stanford Achievement 
Test (SAT). The publishers of these two tests were visited to obtain both technical artd 
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marketing toformation about the tests. The technical and marketing mformation"sought,.as 
well as a report of the findings'from the visits to the test publishers, are discussed belovV. 



Because there will be out-of-level testing in the SES, it will be important to be able to- 
convert a student's score on one level of the test to a scale that covers all levels, in order to 
compare students in a given grade. taking different levels of the test. Vertical scales are 
provided by most test publishers to do just this. However, studies indicated that vertical* 
scales were inadequate when used with disadvantaged children. It w^s thus imperative to 
determine how the test* publishers derived the vertical scales for the two tests under 
consideration, and whether they made any empirical checks on its adequacy. 

Not all achievement test publishers "provide fall and spring norms, based on empirical data. 
Many interpolate norms for periods where no actual data were collected. Projected norms 
are generally based on the assumption of linear cognitive growth over the calendar year. 
Since there is no e\idence to support the linear-growth assumption, the created norms may 
be inaccurate enough to give a distorted view on the i-mpact of CE. The panel on achieve- 
ment tests recommended that, other things being equal, a test should be chosenlhat offers 
empirical fall and spring norms. It had been suggested by the panel that the SES-couIjUji 
itself provide the data necessary to generate better norms than the publishers could supply!* 
A second consideration related to a test's norms is the quality and recency of the norm sam- 
ple If scales based upon t+ie norm safcnple were to be employed in this study, the quality of 
the sample might be a critical consideration. 

Statistical analyses are increasingly being used to detect whether achievement tests are 
biased. In essence, a two-way analysis of variance is carried out, where race is one factor 
and test item is the other factor. If an interaction between race and test item is found," at - 
least one component item is said to be biased. The test is made unbiased by removing test 
items that interact with race (Ozenne, Van Celder, and Cohen, 1974). To perform such an 
analysis, it is important th^t the publisher have indiv idual item data for students of different 
race/ethnicity. 

In choosing the achievement test, the following additional information >had to be deter- 
mined from the test publisher. (1) whether the test could be rapidly scored by a process that 
reads r character Jabels, (2) the royalty rates that applied to test use and scoring, (3) the 
possibility of reprinting selected subtests'of the test in a format more appropriate to the 
study, (4Mhe restrictions involved, arxl permissions needed for reprinting, and (5) the 
royalty rates that applied to the' test printing. 

The visits to the test publishers clarified and revealed t^ie following additional information* 
regarding the^wo contending test Series: 

• Both* publishers were^eager to cooperate in, makirfg both data and computer programs 
available w \ 



u 



Both publishers felt thatlH^ had good normative data. Only the SAT had empirical 
fall and spring norms at^^&e levels, the CTBS had empirical fall and spring norms 
only at the lowe? levels. 



• Both publishers agreed that new empirical norms could be created by re-weighting 
the* study sample Indeed, they encouraged this procedure. 

*• Both publishers use a modification of Jhurstone's method of absolute scaling to 
calibrate various test levels onto a common metric. However, two levels of the test are 
nevet administered to the same group of children in the calibration process. In order 
* to effect the calibration procedure, test publishers made additional statistical assump- 
tions w{>^h may be unwarranted. 

< 

• Vertical scale scores are not available forfait the subtests that would 6e chosen for the 
study. Thus, th^y /riust be created either by the .pfffftsher or from data collected 
during the course of the study. . ' { 

• Only the CTBS publisher collected ethnic data from the standardization sample. The 
availability of such data would .allow for carrying out a de-biasing study at the outset of 
the stfidy rather than waking for the first-year results. 

• Both publishers provided practic^^testT These were used with the standardization 
sample. ^ • J * 

• Both publishers expressed^ willingness t£> create new norms based on the subscales 
chosen. 

Final Test Selection • . 

After considerable scrutiny, it was evident that the SAT and the CTBS were equally good 
representatives of the contemporary achievement tests, however, both had some short- 
comings for the SES. First of all, as complete batteries they take* too long to administer. 
Second, the technical property deemed most important for their use in a longitudinal 
study— the vertical scale— had not been empirically validated or verified. Third, norms 
were not based on a national probability sample. Thus, the SES had to overcome the short- 
comings of the instruments. Consequently, , the selected test would not be u$ed as a 
complete instrument for the measurement of academic achievement, but rather, as a basic 
resource in the ( selection of subscales that CoverVeading and math. The recommendation 
then was that the final selection should be based n6t on technical grounds, but rather on 
pragmatic considerations. . * 

The CTBS was chosen for the SES for the following pragmatic reasons: 
» . * "* 

• It wodld be possible to carry out a de-Biasing itudy based on the pu^lisKveKs data 
m before the first wave ^of data collection, and thus to crq^te revised scoring keys for 

immediate use with the first-year data.- 

• The item layout appeared to be better for the lower levels of- the CTBS than for the 
lower levels Stanford Early School Achievement Test, SESAf ) of the SAT. 
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' * % • The CTBS has been more recently revised than the SAT and thuf probably reflected 
current pedagogical approaches better/ # \ 

• The CTBS publisher had compute^software compatible with that of SDC. 

• The CTBS had been subjected to one round of statistical de-biasing, whereas the SAT 
had not. 

* ** '' 
When the CTBS was selected a$ the standardized achievement measure, several additional 
problems had-to be resolved^in order to implement the test in an effective and responsible 
manner identification and elimination .of biased items; the use of alternate forms; order' 
and learning effects in adm mistering two levels of the test; and level of scores to* be ana- 
lyzed. Two other critical problems, the development of improved norms and vertfcal-scales 
and the effe&s of test speededness are discussed in detail in Report 10 and. in another paper 
on the present report, respectively. 

IDENTIFICATION AND ELIMINATION OF BIASED ITEMS 

The use df standardized achievement tests in previous evaluations of CE has caused 
numerous educators, evaluators, test developers, and rriinQrity group members to express < 
concern that such instruments are not appropriate as measures of academic performance 
for minority students. In the main, criticisms of standardized achievement tests have 
focused on the claim, that such tests were developed for Anglo,, middle-class students and 
are biased against minority students. • 

The Origin, Nature, and Consequences of Sociocultural Bias in Tests. A major concern 
in the SES was {gcused on sociocultural bias in tests. That .is, we were particularly con- 
cerned with biasestesulting from systematic differences in the sociocultural backgrounds 
apd experiences of members of certain groups, where these backgrounds and experiences 
differ substantially from those of other groups. More specifically, we were concerned with 
bias against disadvantaged and minority groups to the extent that the sociocultural history, 
environment, and values of those groups are known to differ from those of the children on 
whom the tests are normed. The consequences of sociocultural test bias depend on how 
the test data are used. All too often/data from biased tests are improperly used to infer 
underlying, innate differences between different racial or cultural groups. Such an inter-* 
pretation may be used to justify 'giving up' on certain segments of the population, i.e., to 
assunfe that it is pointless to make any effort tQ change things that purportedly have inherent 
or hereditary origins. Bias in tests may also influence decisis about who should .be pro- 
moted in school/assigned to advanced learning groups, givenfothertsigns of recognition 
such as membership in ctubs and'sooieties, accepted into coUgges, and hired in desirable 
jobs Use pf biased Jests may produce systematic underrepresentatiorV of disadvantaged 
and minority students in the benefit of these forms of recognition. 

^Finally, and most relevant to the SES, use of a biased 'test may lead one seriously to 
underestimate the value of a CE program sucb as Title I. Such a test could underestimate, 
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for disadvantaged and minority students, not only the absolute achievement but also the 
'gain in achievement that is used as a criterion of program success. 

N^iiven thpse concerns, we conducted a study of the potential] bias in the CTBS. Results of 
•the examination of the CTBS. for biased items would allowj us to perform subsequent 
artalyse^\4ith some certainty that "data will not be overly influenced* by test items that seem 
culturally biased against particular ethnic groups. Review *]of the CTBS for bias was 
accomplished in two Rhase$/ Briefly, the first phase involved tHe identification (litems that 
are statistically biased* against ethnic groAps as demonstrate by the publisher's standard-, 
ization dat^. Ir\jhe second phase/a ten-member panel of persons from a ranpe of ethnic' 
gr«oi4os evaluated the statistically biased items to determine wihicfTtontamed content that 
was^jkjraHy biased ^agaijist' certain ethnic groups. The panel members' review of item 
content ^^carned 6^ by using * a modified' Delphi approach, with two cycles of 
item review^ , ^Jk t ' v 

Statistical Analyse* to Id&qtify. BiasedTtems. T+ie statistical procedures enxptoyed in identify- 
HQg'poterfcially biased keiV^ere based upon the notion that items that appfeaj to 'work " 
differently f(Jr different groups are litfely to be biased. Several approaches to tRe operation- 
aluation^bf 'work differently,' hav^6een utilized in previous studies attempting*to determine 

U4) 'ijjtilizfed two statistical- 
analysis approaches, coupled with *profe^sibr&l jucjgment, m the identification of item bias ' 
in the California Achievement Tests. Their approatjj^s are described below. , 

The first -statistical approach can pe labeled an 'analysis-of-vanance' approach^ the idtnti- 
^ifation of potentially byased items. This Approach is based upon the work «of Cleary and 
Hilton (1968) Iterrv response data, are analyzed within a two-way" factorial model* for 
analysis of variance, with group membership (ethnicity or race) one of the ways, and items 
the other way In this type of apa[ysis of variance, there will be 'two ma[n effects and one 
interaction Evaluation of the main v effects would indicate whetrjgr the ethnic groups are 
different in overall test level arfd whether the items are differewW, difficulty.' The mam 
effects are, for this type of analysis, overlooked in favor of.the interaction effect. A signifi^* 
cant interaction indicates that some (at least one} items ^reXvorkmg differently* from the 
way mO^t others' work. The interaction effect takes into account th&t the ethnic groups may 
b£ different in overall level and that the items may jnave different difficulty levels, 'and then 
looks. for items that are working in ways not expected frpm these main effects. 

Because this analysis-of-vanance design is a repeated-measure design, and because the * 
number of'items analyzed f<^r* any, one test or subtest^relatively -large, a rigorous statistical 
interpretaion of any significant interacticJh is difficult t© make. The problems, of course, are 
in accounting for the correlated nature of the item respcyises (the same students answer all 
the items) and then identifying, through some .s6rt of after-the-fact tests, precisely which ^ 
items are causing'the significant findings. The rigorous statistical identification of items was < 
av oided by Qzenne et al , by substituting graphic methods for detecting thoSe items th£t 
were 'working' differently, ' 

\ rf - * " ? 

*lt should be noted th^t the correspondence between statibticdl bias' and bids ds usually meant in terms of 
justice or morality ^ based on a nunYber ol inferences that are not compelling to all researchers who study # 

• the problem * ( • ' f (/ ' 

• . . ' * • . 173'*' ^ ~ ' 



For the SES.a two-factor design w.ith repeated measures on one factor (Winer, '1971) was 
employed in the analysis of each skill area and test level. Subtests within skill areas were 
analyzed jointly, as the total skill-area scores were* to be employed in the study, and it is 
these total„scores that should be free from bras. Analysis of each of the subtests separately 
might fail to identify items that are biased in part because of the bias of the subtest of which 
they are part For each tesrle\el, one grade of btudenrts was selected. 'Selectioli of the one 
grade' per level trbt provided -the largest number of students 'sharpens' the analyses by not 
allowing between-grade variances to enter into the comparisons. Because in tlje 
publisher's norm sample rhe ethnic grouping of brown studfafc was always thelsmallest in 
-number ar*d because ,the computer program (BMD 08V)" calls fof an equal number 6f cases 
- in each cell, the grade level for'the analysis of each test level was selected to makimize the 
number of bcbwn students^he number of brown students was then matched by| randomly 
selecting equal numbers of'Black and other students. In this way, sample students from' 
each ethnic group were selected for the following- test levels. 222 from kindergarten for 
Level A; 318 from grade 1 for-Level B^231 from grade 2 for .Level C; 382 from gAide 3 for 
f Level 1 457 rrom grade 5 for Level 2. 999. mom grade 7 for'ievel 3, and 717 from grbde 9 for 
Level 4 Summary tables from the two-factor" repeated measures analyses of vartlnce are 
provided- in Table 9-1 V> / • 

Inspection or Table 9-1 indicates that in each'of 14 analyses, both of'lhe main effects and 
4 t*he*interaction are significant The interpretation of the ethqipty' rnain effect, as provided 
by the overall means, is that in all cases, the black students 'earned tbe lowest mean score 
and the other students earned the highest mean scoj-e. (One must, of course, guard against 
any obvious interpretation of these cpnsrstent findings, .as* ethnicityAm this samplers 
undoubtedly co-founded y\ith a. host of socioeconomic, and cultural variables'}* The 
significance of the tfem' matn effect, indicates thaUhe items have a wide! range of difficulty 
levels In general, within a separately timed subtest, the early^ ferns are relatively easy, 
while the late items are relatively '"dvfncult The* significant interaction terms innate that 
some* items are differentially difficult over *h~e'thre.e groups. It is these items that were 
identified as statistical^ biased. 

The method employed to* determine specifically which of theitems are so biased is a 
variant or the method Winer (1.971. pp.^2§ ; 53h called a t<>st of simple pam effects, In the 
variant method employe*, hawever/the main effects 1 are ignored and the simple -interac- 
tion .fftect is studied iThis is accomplished by analyzing e3eh item separately over the three 
groups, as in a simple I x 3 analvsis of variance. The difference is that the overall'group- 
,errect mean is subtracted from each subject's responses within each group, thus eliminating 
the mam effect (^ethnicity, and the error.term, the denominator or the tekt, is an estimate 

. or jhe pooled-wtthHjsum of squares, taken as an average over all the items. The error 
'estimate is thenlempj&yed as the dehominator for ealth of the item\s»mple-intefaction- 
effect; tests. The resulting F ratio does not hav^an F distribution, but its d^nbution may be - 
approximated another F disfribCTtion with different degrees 6r freedom (Winer/ 1971, pp. 
530-5}f ) equal to one less than the number tif^roups and (as a consertative-test estimate> 

% the product of the number pf groups c^d one less than the'number oi replications. The 
conservative estimate was selected because of its simplicity and because the large sample 

^Jaumbti o^refjlicatior^) provided r^r degrees of freedom greater rfhan those typicalTy 
tabulated anyway. * 

* * 



Table 9-1 ^ \ 
Repeated-Measures Analyses of Variance fdr. Seven Levels of CTBS Skill-Area Scores 
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These tests were systematically applied to each item within each skill area at each level. 
* Although the ideal method for the evaluation of the resultant 'single-factor test would be to 
predetermine the acceptable Type II error (accepting the hypothesis that a truly biased item 
is not bhaseclL the sample sizes within cells were sufficiently large that the more convenient 
method of pre-setting an alpha would hot likeVcause any differences in the identification 
of potentially bjased items These analyses, alpha was conservatively set at .001, partly 
because of the questionable assumptions made on the covan<ince ny/tnces and partly 
'because the resulting F statistic; does not truly have an F distribution^Si^ty-seven items with 
significant F tests are listed in Table 9-2, with their respective subtfSTcocjes and the original 
. itefrv numbers from Form S of the CTBS. 

It can be seen from Table 9-2 that, as in the study by Ozenne etal. (1974), there are fewer 
items identified at the lower levels than at the higher levels, and not all the items appear to 
be biased against the minority groups. Among the 36 identified reading items, 4 appear 
biased against black students, 8 against browr||!5fodents, 12 against other students, 10 
against both black and brown students, and 2 against brown and other students. Among 
the 31 identified math items, 12 appear biased against black* students, 2,'agamst brown 
students 14 against other students, 2 against black and brown stude/ffsSind 1 against black 
and other students Inspection of Table 9-2 will also reveal that there is not a consistent 
trend for early (easy) or late (hard) items to be biased against any one or two groups. If the 
analysis-of-vanance approach to the identification of biased items is accepted as e mean- 
ingful way to study tesrbias, then these results indicate that the CTBS is fairly evenly biased 
against all three groups 

The second statistical approach can be labeled a 'correlational' approach to the identifica- 
tion of potentially biased items This approach has'bee/i utilized by several test publishers 
in their efforts to eliminate items giving the appearance o^bemg biasedAn this approach, 
Ttefns'are analyzed through the correlation of item-response pattern to total^scpce-pattern 
within each racial'ethmc group, so that it^ms having different correlational levels among, 
the groups can be assumed to be measuring different things via the notion of mternal- 
^misfency reliability Indeed, this correlational approach ts equivalent to t£e examination 

*bf differences among the item-total bisenal correlation coefficients 'for each* of "the 
racial'etjimc grdups The bisenal coefficient should be employed because of its relative 
resistance to effects of the general difficulty Hevel upon its size (Oosterhof, 1976LBecau5e 

. of dffficDfpTes m the estimation of the standard errors of differences between bisefcjal coeffi- N 
ci^nts however the analyses were performed upon pomt-bisenal- coefficients. Each triplet 
of point bisenals car> then be tes'ted for each item to determine if two of the coefficients are . 
statistically different across'the racial/ethnic groups (the intercTorrelation among test items, 
the repeated metres aspect of the first type of analyses, cannot be^considered in these 
simple univariate tests) .Ozenne et al^ag^iin 'utilized'- a judgmental approach to the 
identification of those items that seemed to. have different item-total correlations for th^ir 
different groups 

In thySES. pomt-bisenaL coefficients between items and their totals werfc computed for* 
^eacj/ r,K i a Tefhnic group formal! subtest within skjl, areas. The method for determining 
r * Sfriether or not the coefficients were different was <j simplified test«based upon the point-! 
t * • t • • 

" « , • . ^ % ' ' \ 

"\ , m \ \ 
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CTBS Ifewisidentified as Biased by the Analysis of Variance 
^0B(pach, and Group(s) Biased Against 
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bisenal's essential identity to the Pearson r (a characteristic not shared by the bfsenal coeffi- 
cient), and thereby to its sampling characteristics/ The point-bisenals were -tested in the 
followmg-m'anner the largest and smallest coefficients for each of the 88/ items were 
trapsformedjnto Fisher 2r$, and then a z test was made of their differences (Guilford, 
196^). Because of the Jarge sample sizes involved in these tests, alpha was set at .001. 

Of the 88" triplets of coefficientslhus tested/ 155 exhibited significant differences. Table 9-3 
presents tbe 155 item numbers, according to test level, skill area, and subtest, that were 
identified as biased by this correlation! method, the results of the Correlation analyses are 
similar to those of the analyses of variance, as they both^intTicate more biased items at the 
higher levels than at the lower levels These data are unable to address whether this 



Table 9-3 

CTBS Items Identified as Biased by tjie Correlational Approach 
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phenomenon is due to large samples at the higher levels, or to some sort of withm-group 
homogeneity and between-group heterogeneity that develops with age. 

AlHhe items that were associated with significance, either in the analysis-of-variance o.r the 
point-bisenal examinations, were considered .'suspicious' items that would undergo 
judgmental analysis A totirof 201 items (67 identified by analysis of variance, plus 155 
identified by' correlational analysis, minus 21 that were identified both ways)* were sub- 
mitted to the bias panel, along with 1 4 -additional items (one each for reading and math at 
each of seven test levels) that were identified as being statistically least biased (minimum 
observed differences in the statistical analyse^. The unbiased items were included in or^er 
to support a caution to the judges that^fct all items had ; be£h identified as biased. 

4. * 

'Judgment Analyse^ to Identify Biasec( Items. The items identified by the above procedures 
were suspected of bias because of their differing statistical properties in different subgroups. 
However, the statistical analysis does not indicate exactly what property o(an item makes it 
biased. In fact, the statistical analysis may identify many item$ that are not biased but that 
have aberrant statistical characteristics only Since the objective is to eliminate items that 
are truly biased against any of the subgroups involved m the study, it was necessary to iden- 
tify the source of p6tential bias m each item before removing it from its test. A review of all 
the btanstically biased items was therefore conducted to determine whether each item hacJ 
a content o^form that could bias it against one qr more of the ethnic-cultural subgroups 
participating in the SES. . » 

To evaluate the items determined to be statistically biased, a pane^jf ten people irom^ 
variety of ethnic backgrounds was used.- *Astan Panelists. From Hawaii, an elementary 
administrator and district Title \ Director from CaWernia, an urb^n elementary teacher. 
Black Panelists. From Mississippi, an expert m item construction and test development, 
and director of testing at a predommatly black univer$fty, from California, an experienced 
metropolitan elementary teacher, from South Carolina, an experienced urban, elementary 
teacher and Title I coordinator^ Chicano Panelists. From Texas, an expert in item 
construction and test development wh<!> had experience teaching Chicano stu.dents, from 
California, a community leader. Irfdian Panelist. From New Mexico, an expert in item 
construction and Head Start Teacher of Indian Students. White Panelists. Frpm'West 
Virginia, *an .experienced elerrjentary and special education teacher of Appalachian 
students, fro.m Massachusetts, anieducationaf.desegregation specialist/ 

The panelists carried out the*V review of the content of statically biased items following a 
m6dified Delphi approach! Two separate r§view$of item/ were made. In the first Feview,^ 
panelists were sent copied of all levels of the CTBS along with instructions for their review. 
In each of the, test booklets, items to br reviewed were marked (and, for the lower levels, 
directions to *be read by the test administrator were included), and the instructions 
indicated that the panelist was to rate each item as either biased,' possibly. biased/ or 'not 
biased' on a rating form. No item was rated s biased' or possibly biased' by more than six 
panelists/On the other hand, 99 ol^the 215 iteni^were'so /ated by at least °one panefist. A 
conservative criterion was G$ed in determining which iteraS vyere to be considered again 
during the second review, that of including all items rated biased' or 'possibly* biased" by.at 
least two panel me/nbers. Thirty-two items met this criterion. ♦ 
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For -the second review: panelists were "sent forms for each of the 32 items, along with 
comments made by panels in the first review; and were asked to rate the items as either 
'biased' or 'not biased ' Panelists were instructed to note which commenl they subscribed 
tq/and to include additronal comments for any item that was teted 'biased.' x 

In order for an item finally to be characterised as being biased, criteria were established 
that at least five of the panelists had to rate the item as biased in the second review, and the 
item had to be statistically biased (the latter is important, since two^of the 'control' items 
were rated biased by five or more panelists). These criteria resulted \njhe identification of 
seven items a* culturally biased. One additional item was determmed*to be biased which 
did not meet the above criteria, it is an item that was statistically biased against -brown 
students and the two Chicano panel members indicated that the key wo/d used in the item 
stein' had a Spanish root that would" apply equally to two of the response alternatives. Thus, 
a total of eight items were identified as being both statistically ^nd culturally biased. 

All of the eight biased items come from .reading subtests, none of the math subtest items 
were, judged to be biased Grouped by test level characteristics of the biased items are 
indicated below 

Lewi 4- Two it.erhs, from the Sound Matching' pprtion of the reading subtest. Both 
items require the student to indicate whether two words, said aloud by the examiner, 
are the same or different. -For each item, .seven panelists noted that children from 
certain ethnic groups are unfamiliar with the differences between the critical sounds 
in the paired words. 

Level ff-'No biased^ems . * 

Level C One item, from the/heading Comprehension, Passages' portion of the test. 
Seven p^elists commuted (hat the setting of the passage and the. items of importance 
m the passage araajpt within the 'scope of experience of most minority children. 



Level 1 Tvv^ftems, one from the 'Reading Vocabulary' and one from the 'Reading * 
Comprehension' portions of the test. The vocabulary items asked about a word that 
children of different ethnic groups would not experience often. The comprehension 
- e ^li :onla i Q ^ d a xtUn i unfamiliar to many minority children. . < 

4 • - • 

Level 2 No' biased items 

Level 3' Three items, two* from the 'Reading Vocabulary' portion and one frorti ttfe 
* 'Reading Com P rehens,on ' portion of the test. One Vocabulary item dealt with'a word 

considered outside the vocabulacy of most v bjack children, the other vocabulary .item 
4 is the one mentioned earlier that could mislead students from. Spanfch-speakmg 

backgrounds "because of the Spanish root word. The comprehension item turned on a 

particular word judge.d to he an Anglo expressioh, and was based on a passage with a 

locale to which many minority children would not be exposed. 

Level 4. No biased items. 



After the biased items were identified, the data tapes provided by the publisher fof the 
statistical de-biasing were rescored to determine the characteristics of the d^biased scores 
N and how they related to the no/vde-biased scores. The results of this final analysis are 
presented in Table 9-4.' 

, Tvk) notes of explanation appear necessary to Conclude this description of the nee^T 
^ methods, and results of the de-biasing efforts of the CTBS. First, we must address the fact 
x that only eight items* (out of a population 'of 887 items; were ultimately identified as being 
biased. It may be concluded that this low incidence is du^ to the effQrts of the publisher of 
tVie CTBS, first to develop* tests that are minimally biased, and second to its own statistical 
d^-biasmg »n order to weed out items that eluded the developers. It should be pointed out 
thaj at no time during, the de-biasing process did thre SDC team set compromising criteria 
,for\he identification of biased items, jndeed, all the criteria were set prior to 'the study and 
' appeared to be reasonable. 



Table 9-4 



Comparison of Non-De-Biased and De-Biased CTBS Reading Scores 



TesJ^4evel \ Non-De- Biased 

Number and Ethnicity of Students Mean 



De-Biased 



S.D. 



Mean, 



S.D. 



Level A * 
666 fondergarten students 
222 Black 
222 8>own 
222 Other 

Level ' 
693 Second-grade students 
231 Black 

2>-B<cnv9« •„ * - 
2t\ Other/ / 

? » * * 
Level I 

A 146 Third-grade students 
332 Black ^ ■ 
332 Brown 
382 Othe r ' 

Level 3 I 

2 997 Seventh-grade indents 
* 999 B*lack 

^ 999 Brown ' ^ 

W Other 



32 92 


10 31 


31 50 


9 78 


998* 


29 24 


1040 


27 95 


9 78 . 


9982 


31 36 


9 27 


30 03 


880 


9976 


38 17 


9 05 


36 51 


8 63 


9985 


33 65' £ 


1*43 


32 92 


12 18 


9996 


29 52 


12.75 


28 88 


12 47 


9995 


32 81 ' 


11.71 


32 06 


11 49 


9995 


38 62 


11 08 


37 83 


10.81 


• 9996 


i 

41 60 


19 19 


40 73 


18,72 


9995 


34 66 ^ 


16.58 


33 91 


16.20 


.9993 


36 83 


16 49 


36 68 


16 To 


9992 


53 31 


18 75 ' 


52 21 


18 18 


9995 


42 19 


1840 ^ 


40 59 


1768 


9992 


34 39 


15 52 


33 20 . 


•14 92 


9988 




15 87 


36*16 


15 19 


.9989 


54 52 


17 05 


52 42 


1646 


9991 
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The second explanafoty note is concerned with 'the relatively small mean differences in test 
scores resulting from^the de-biasing effort and the high interrelation between the 
de-biased^nd non de-biased scores. Noting these (not unexpected) small changes, one 

..might ask* what was gained from the de-biasing effort. The statistics in Table 9-4 do not 
indicate how scores for different subgroups of students will be influenced, hfljw growth 
indexes will be influenced, or how the significance and meaning of statistical comparisons 

, will be influenced With the elimination of the biased items, however, we can be confident 
that however large the influences are, they will be in the-direction of providing conclusions 
that will be fairer to mmo'rity and disadvantaged children. 

THE US€ Of ALTERNATE FORMS . , 

Early in the pfannmg'of the schedule oftests, the' Pane) on Achievement Tests recom- 
mended that alternate forms of the tests' not be employed in the SES. The panel was most 
concerned witlfThe continuity of measurement over the years of the study. If longitudinal 
conclusions are to be drawn, then efforts must be made to ensure that similar aspects of the 
growth dimension are measured at each assessment point The utilization of alternate forms 
of the achievement test while addressing concerns that mere practice effects^ many 
account tor findings also reduces the continuity of the measured outcomes. Empirical 
observations of alternate-form reliabilities of standardized achievement tests indicate that 
they tend to be about 10 lower than their corresponding internal-consistency or tesf-retest 
coefficients, Over a period of years with several retestings, tf^>elatively small inconsisten- 
cies may influence the Interpretation of the findings. Levme and Angoff (1958) present 
evidence supporting the conclusion that pwctice effects are generally smaller than 
inconsistency' eTfects jn test-retest studies. " 

i 

m 

\ more pragmatic reason for utilizing onl> one form of the achievement test involved the 
naed to create a new vertical scale of growth fhe scaling procedures that are employed in 
the development of the vertical scale are highly dependent for their accuracy upon col- 
Imeantv of the test levels being,scaled Alternating test forms over the levels reduces the 
•collmeantv In addition, to the extent that the forms are not collinear, they will create dif- 
ferent expansions or, contractions within the scale. In this event, either different vertical 
scales would have to be constructed for the various sequences of forms, or the -vertical 
scale would JiAe. to be 'smooth' over those differences. Based 'upon these considerations, 
we decided to employ only ForrmS of the CTBS for all test'admi^st rat ions, 

ORDER AND LEARNING EFFECT* IN ADMINISTERING TWO LEVELS OF THE TEST 

During the first year of the SES, each participating student was to be administered two levels 
of the standardized achievement test, a grade-appropriate lat-levelf test and a test one-level 
below grade-appropriat? f befow-le*vel). This procedure is employed for two reasons. First, 
double testing should make possible the development of a highly accurate vertical scale of 
growth 'see Report 10) Second, it will be possible to determine the best functional leveHtff* 
testmg for each grade level at each school so that testing during the/second year and 
"beyond can be at an optimal level for measurement. It is not unreasonable to expect, with a 
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large representation of economically disadvantaged, schools arid students, that about half 
the schools participating in the study will employ below-level Jesting during the second 
year (see Hoepfner and ; Wang, this volume). ; . 

Where there is double testing there is reason to^ belie/ze that some phenomenon will 
influence scores on the second test that would not'have7nfluenced them ha^ti'the second • 
test been administered first. If, for e^mpley^secpnd-test scores are depressed, we might 
expect fatigue or boredom 40 have'playerf a role. If second-test scores are enhance**, 
perhaps learning or sonpe sort of transfer has occurred. In more general terms, either out- 
. come would indicate that the order in which a test is taken will influence the score earned. 
In the^ES, the ordefof test administration should be determined to minimize the impact of 
the orderjeffects upon the validity, acceptability, and logistics of the study. ■ „ 

Frorti the validity standpoint, order effects had to be minimized for the at'Ievel test, because 
those scores, converted to publisher-provided norms, were to be returned to the schools 
for their use. Order effects had alsb to be minimized so that they not be built into .the 
vertical scale, thu* giving an inaccurate picture of growth over levels of the test. From the 

^>oint of view of-aoceptability. the sequence of tesf administered wduld have to be justified 
to school personnel, especially if it was the sequence that does not hive an obvious 

/rationale The loWics of the study are affected only by a decision to counterbalance— to 
employ different te^dmmistration orders at different classrooms, grade levels, or schools - — 
Clearly, si£ h differences complicate thfc packaging and record-keeping procedures in 
addition to the training of coordinators and classroom teachers. Actually, th^co/nphcations 
caused ^by counterbalancing test order are sufficiently grave to render such a decision very 
undesirable £rom a logistical point of view. 

The planning-year field test of the SES was, therefore, in part an effort to determine tfie test 
order(s) to be adopted for the first-year test administrations. .Four schools each, within each * 
of four categories, were selected for the field test. Northeast-urban-poor, Southeast-rural- 
poor, Southwest-urban-non-poor, and Nerthwest-rural-non-poor.-At each school selected, 
two dSssrooms atjjfades 2, 4, and 6 were administered the achievement test; two 
randomly-assigned schools in each-cell being tested with two reading tests, and two with 
two math tests Within each grade level of each school, one randomly assigned classroom 
was administered the below-level test first and the other was administered the at-level test 
first Th'is balanced field test plan was designed to yield information on the effects of order 
upon the test "scores. The tests were administered in a standardized way under the observa- 
tion of a trained SES coordinator. The scores were converted to percentiles for ease of* 
comparispn, total score for reading and computation scale score for*math. Table 9-5 
presents the mean percentiles and the numbers of participating ^examinees under each 
testing condition for reading and for math. \ ' * 

Analyses of variance within each grade level for reading and /or rnath revealed significant 
order effects in all cases. Statistical significance, however, must in this instance be inter- 
preted in ligjit of the meaningful ness (quantity) of effect observed. Table 9-5 was supplied 
for this purpose. ' ♦ . 
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Analysis of Order Effects on Reading and Math Percentile Scores 
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Reading 






: 4 
6 


'178 
185 
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50.44 
- < 42 42 


59.16 
44.67 
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181* 

193 


54.66 
45 81 
♦49.05 * 


48.37 
44.75 „ 
46.93 1 
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s 


559 


47 43 • 
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C A Q 

54o 


v 49.76' 


46.67 
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Math 








* 189 
f 215 # 
* 214 


37 65 
49 95 
55 26 


42.01 
50 08 
50.92 


206 
' 218 

. 2,3 ./ 


45.97 
60.64 
' . 51.88. 


50,82 • • 
57.48 ^ 
55.54 ' . 


* 

. All 


' 618 


, 48.03 / 


' 47.90 


_637 5 


52.00^ 


54.68 . 



Comparing mean reading percentiles for the same level over the different orders, one can 
detect small' and inconsistent differences. The largest difference is 4.5 percentiles^ the 
at-level test, higher ^when administered second.- Over all grades, it can be seen that the 
'below-level- test mean is reduced by J6 percentile when \{ is administered second, while 
the-at-fevel test mean is increased by :52 percentile. The average difference in percentile 

^means-between the two levels is 4.19 percentiles in the belbw-ther>at order, and 3.09 
percentiles in the at-then-below order. Considering the rather large standard, error for 

^percentile CQnyerSions, these differences may all be meaningless. "From these summary 
findings it would appear fhat the reading, test could safely be administered ih any order. 

Comparing" mean math percentiles for the same level over different orrjers, the differences 
Sfcem consistently larger, although just as mconststent. The largest difference in Table 9-5 is- 
13.17 percentiles for the beiovy-level test, higher when administered <£cond, Over all 
Vgfsteles it canbe seen that the below-level test'mean is increased by 6*65 percentile when it 
f is administered second, white at the af-jevel test mean is reduced by 4.10 percentiles. The 
average difference in percertnle means between the tvyo levels is 2.88 percentiles in the^ 
below-then-at order, and 3.89 percentiles in the at-then-Jselow order. >! 

• , ..-.•-«- , . y 

Because'these dat&do' not provide an unequivocal answer to the question regarding order 
-of test administration, irwas decided to consider eiich order separately to see what advan- 
tages and disadvantages would resuft. Weighjng^the advantages and disadvantages in each 
test order, it was decided to test in a constant below-then«at order. This order would meet 
with the greatest school personnel approval, wdtjld reduce test trauma in first-graders, and 
wogJd result in very few and very small technical advantages. 



" LEVEL OF SCORES TO BE ANAfe¥ZE^ * 

Administration of the achievement tests can result in several different types of scores. In the 
case of the CTBS,, several varieties of types of scores can be considered. As examples,, one 
might wish to study total achievement scores, skill-area achievement scores, sub-scale 
scores, or even item scores. We elected to employ skill-area scores (reading total and rftath 
1 'total) rather than more general or more constituent scores. This decision was made on^the 
basis of three .considerations, acceptability,, vertical-scale construction, and colljnearity 
among dependent variables. - 

; Score Acceptability. On the basis of conversations with many educators* it was concluded 
that few if any teachers or school principals would be concerned that the study employed 
^ skill-area scutes rather than total .or sub-scale scores. Although instruction is presurr&blyijgr- . 
geted at more specific Jevels, its effects spread across specific learning objectives. Reading 
teachers, and most materials used to teach reading, stress both votabulary c building and 
compression improvement— most frequently together, in math the stresses on concepts 
^largely as an aid to computation This argument Confirmed us in our'decision not to observe 
*the traditional approach, but to analyse achievement scores at the hghfcr ^nd more easily 
'understood conceptual -level. 



Vertical Scj/e Con^truiAion. The develc^am^^^ade^tuate iertical scales reqqires4faat4he 
* achievement -test data Collected meet some basic assumptions* ' • 

That the scores have a sufficiently fine metric such that many gradations are possible: 
fbe more gradations, the more accurate the intra-teyel matching can be. 

| That the scores over the entire vertical range are Tineariy/ related? i.e., the scores 
* measure the same thing or the same collection of thing5. ' i "* 

That the scores are reliable measures of what theyVieasure, so that scale extrapola- 
^ tions from scores can be made with confidence' that.tTie score belongs where it ♦is 5 ' 

placed on* thfc. vertical scale,* ~ ■ A 

We studied the publisher's data to see if the tise of subtest scores could be justified and, if 
£ So, what' problems might arise that would be dangerpys to the conclusions of the study. 
Table* 9-6'and 9-7 provide the scores and Renumber of ntems for each score that would be 
used under^kil&uea total and und£r sub-scale score conditions. - 

The scaling ot skill-area total scores wilf be fairly straightforward. The number of items 
- (gradations of metncj is large, and the even columns indicate {hat conceptually '(at least) 
there is commonness of content (linearity). - ] 

Table 9-7 presents quite a different picture. The numb^of, items of some .of the sui^cale 
bcores is very small, setting severe limitations upon the accuracy of any vertical scaleWiat is 
built upon them. Further, the uneven columns indicate that there are problems as tt) which 
scores would be scaled to which other scores ^"different levels. Fop example, while there 
appear to be no problems % at and above Level l£3t the early fevels we would have,4o 
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Table 9-6 

Vertical Scales if Skill-Area Totals Are Used 



Levef 


Reading Totals 


«-» 


Math Totals 


A 


Reading tbtal- 49 items 




Math totaL 26 items 




Reading total 43 items 


; 


Math total: 46 items 




Reading total- Sfitems * 




Math totah 40 items 




/ ' Readioglotal 85 items ' 




Math total" 73 items 




Readin^Jtotal £5 items 




Math total: 73 items 


3 


Reading total 85 items 




Math total; 73 items 


* 4 


Reading total 85 items x < 




Math total: 73 items 



Table 9-7 

Vertical Scales if Sub-Scale .Scores Are Used 



Test 
Level 




Sub-Scales / * * 




, r 


Reading 3 a ' 


A 

B " 


Sound Matching 28 items 
' Word Recognition II- 19 items 


I Letter Sounds: 0 21 iterosr 

, i Reading Comprehension: 24 items 


C 


Vocabulary 33 items 


Comprehension Passages. 18 items 


i ' 


Vocabulary 40 items 


Comprehension. 45 items * 


2 


Vocabulary,, 40 items 


Comprehension 1 45 items 


3 


Vocabulary 40 items * 


.Comprehension' 45 items 


4 


Vocabulary; 40 rtems 


Comprehension 45 items 






Math c 


A. a,* 


. Mathematics. 26 Items 


B 


Concepts 14 N }lems 


Computation- 32 'items 


C 


Concepts 12 items £ 


Computation 1 28 items 

/ v 


r - 


Concepts 25 items 

* 


Computation: 48 items 


* 2 
3 


Concepts. 25 items 
^ Concepts* 25' items ^ 


* 0 Computation 48 items 
Computation, 48 items 


4 


t Concepts- 25 items 


Computation- 48 items 
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choose,* according to some criterion! (tmkrWn), which of theV^jkfeg. sub-scales belongs 
to the^vodrbUlary' vertical, scale* and Wiich totJ^compre^sW^vertical scale. The 
problem is even worse when one observes trurTnthe math area there is only one short test 
at ievel A; to which vertical sc/e, concepts' or 'computation,' does it get assigned?" 
(It -shouldn't be assigne^ to both, because findings^will be totally colhnear -then, and 
it Jhe scale is split, there will be so few- items^that'the vertical scale will be exceedingly 
unreliably constructed.) x ' 

% 

% ' ■ I *■ 

Of course, the continuity' of the vertical scales cannot be judged solely on the -basis of 

the names attached to the soales For that reason Table 9-8 presents the empirical mtercor- 
relations between scales in Levels 1 through 4, using both skill-area scores and sub- 
scale scores. .% ' , « • 

Notice that mTable 9-8. even disregarding the problems of linearity between levels at the 
lower grades, the inter-correlations between levels using s^ill-area scores are consistently 
higher generally by an average of about +.045. Keep in mind that this small difference 
does not include any differences at the crrtical lower levels. (The publisher did not provide 
intralevel inter-correlations because of the difficulty of matching; use of the sub-scale scores 
would compound, thisp^iculty ) ' * ' - 

Another way of evaluating the effectiveness of skill-area vs. sub-scale total scores would be 
tolopk at their internal reliabilities. These reliability estimates aVprovided in Table 9-9. 

The average internal reliability for the skill-area totals is .9229; wbile4hat for the sub-scale 
totals is 8959, for an average difference of + .0270 in favor of the skill-area total scores/' ■ 

A closely related approach to the relative evaluation of the score types involves inspection 
of the standard -errors of measurement (SEMsj of the scores. The SEMs reported in Table . 



Table 9-8 



Intra-Level Intercorrelations Among Scales 

















Test Levels. 






Skill-Anga Total Scored " 








Reading 






Math 




! and 2 * 
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. 75 
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-> 


2 and ) 
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87 






78 




3 and 4 




*84 






.83 


. / - 
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Sub-Scale Total Scores 






— 35 


Vocabulary 


Compret^sion 


Concep 




Computation 


a. ! and 2 


79 




8^ 


, 72 




' .75* 


2 and 3 


76 




81 * 


77 




78 


3 and 4 


79 
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76 


. 76 




83 . 



Table 9-9 



1 


Internal Reliabilities of Scales 


y 


T$st Vevei 


Skill-Area Total Scores 


f 




Reading 

■ ~- • 


Math 


A 
B 
C 

r 


.91 
93 


.81 
90 
.89 


: 2 

. „ # 3 

y 

4 . « 


.96 
96 

.94 ' ' 


.95 
.92 
.94 
. .94 




Sub-Scale Total Scores 






r . Reading 


Math 


V 


Sound Meaning- .3T~~ 
Letter Sounds. 88 


Mathematics .81 


b 


•Word Recognition, II. 87 
Reading Comprehension: .91 


*. 

Concepts: .83 
Computation' .90 * 


c • 


Vocabulary: .93 , * 
• Comprehension. .92 


Concepts: .85 
Computation: .89 




Vocabulary: .92 
Comprehension:, .94. 


Concepts: .89 
Computation: .95 


- 2 


Vocabulary: .92 
* Comprehension: '.92 


Concepts* .82 « 
Computation: .92 


3 


Vocab^ry: .93 1 
•Co/npcehension: .92 % ^ 


Concepts: .85 ^ 
Computation: .94 


4 


Vocabulary: .90 '! 

Comprgkensicm: £0 
! 1 


Concepts: .84 \* 
Computation-, .93 
-A — 1-1 



9-10' are' for .vertical-scale scores, as they presumably have mote- nearly equaL jinits 
of measurements . * • « . * ° . - " * 

Summarizing Table 9-10 in terms of overall averages would be misleading, but it is impor- 
tant to notice that the SEMs in the rightmost column are considerably smaller than those in 
* the left columns in eight of the twelve comparisons. Jhese SEMs^ of course, will underly all 
the ahalyses and interpretations of achievement outcomes in the SES. 

To summarize the concerns regarding the°devefoprrient of' adequate vertical scales, the 
skill-area scores exhibit finer 'gradation, greater reliability/ and greater between-level 
correlations than tio the sub-scale scores. Inadditidn, it is possible to assign scoresjnto the 



Table 9-10 

Standard Errors of Measurement of Skill-Area 
and Sub-Scale Total (Scaled Scores) 
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Level 



Sub-Scales 



Skill 
Area 









Reading t 






A 


Sound Matching: ■ 


3 01 




j . j j 


* 


B 


Word Recognition"!! 


7.72 


Reading Comprehension: 


6.01 


8.33 


C 


Vocabulary: 


1590 


Comprehension Passages: 


9.67 


9.93 


1 


Vocabulary: 


19.77 


f/^mnrphpnCinn* 

V,UI 1 1 yJ\ CI ICI 1. 


1 0 


1 A CiA 


.2 


Vocabulary* 


22 83 


Cnmnrphpn^inrv 

V.UI 1 1 yJl CI ICI IjIVJI 1. 




1 7*^9 
I / !jZ 


3 


Vocabulary- 


* 27.99 


Comprehension: 


31.14 


20 27 


4 


Vocabulary 


34 80 . 


Comprehension. 


37.75 


24.81 








Math 






A 


Mathematics 


13.45 








B 


Concepts- 


11.26. 


Computation: 


12.47 


- 1*2.77 


C 


Concepts: 


11.37 


Computation 


14.59 


13.14 


1 


^ Concepts. 


23 67 


• Computation- 


10:95 


9.80 : 


* 2 


. Concepts. 


34.ai • 


Computation: 


17.84 


14.69 


3 


Concepts: 


37 68 * 


Computation: 


20.59 ' 


16.90 


4 


Concepts: 
' 


44 28 » 

A 


Computation: 


25.94 * 


21.20 



vertical scales very unambiguously in the case of the skilj ^reas. At the lower test levels 
there is no obvious route to assign sub-scale scores to vertical scales. 

Collinearity of the Dependent Variables. It is probably true that all dependent variables 
in the SES will be collinear to some degree. It was our desire, however,, to select measures 
that would be minimally collineap.*The reason for this, of course, is that with non-collinear 
dependent variables, all findings can be directly interpreted; while with collinear ones, 
discrepancies between findings cause problems. The best indication of the collinearity to 
be expected is to examine the intercorrelation* between the dependent variables wheA 
subhscale total scores are used., These* intercorrelations, both raw and corrected for 
unreliability, are reported in Table 901. * 



The average intercorrelation, corrected for unreliability of the component scales, is .81. 
This means/that 66 percent of jthe reliable variance of any of the dependent variables would 
be shared by its parallel dep^ndent^ variable, on the average. SChe problem with such col- 
linearity is not that the analyses are alfclose to duplicates of eachother, but that when 
differences are observed* between two paired deperxJent variables, the chances are that the 
differences are artifactual and should not be interpreted. In this event, of course,, the 
c&ntention that the study will provide information. that can, be interpreted differently for 
each subject-matter component causes it to lose considerable credibility. * 



eric 



168 



194 



Table 9-11 



Raw 3nd Corrected Intercorrelations Between Reading Sub-Scale 
and Math Sub-Scale Total Scores as Indexes of Collinearity 



I est 
Level 


* pending Intercorrelations 


Math Intercorrelations 


" Raw, 


, Corrected 


Raw 


Corrected 


' A 


.51 


57 




! - 


4 B 


74 


83 


70—- 


81 


C 


62 


. * .67 


59 


,68 


* 


83, " 


.89 




,79 


2 


.82 


^ 89 


69 


,79 


3 


84 


91 


79 


,88 


4 


82 


91 


' .80 


91 



The argument for the use of total scores rather than sub-scale scores is strengthened by pro- 
viding empirical information on ^Te. col linearity not only ? of the sub-scale scores but also of 
the 'gain' scores derived from the sub-scales. Table 9-12 f provides critical information based 
on second-year data from the ESAA study (Ozenne, 19^6). The correlations between the 
various sub-scale gain scores are considerably Ibwer than those between' the sub-scale 
scores. This should not be considered as arguing against the collinearity of 'the gain scores 
because the reliability of those gams is very questionable. (Theoretically, the gams are 
eqya! to the posttest scores mfnus all the true variance that they 'shared with the pretest 
scores, meaning that their true variance has been reduced mu^ more than their error 

1 , « ' . ' Table 9-12 ■ 

N - : ' • \ 

e Intercorrelations (Collinearity) Between Syb-Scale Raw Scores* 
* and Raw Sub-Scale Cain Scores**- 



Grade 




<• 

Reading 


n » 


Math 


Sub-Scale 


Sub-Scale Gain 


Sub-Scale 


Sub-Scale Gain 


* *4 


814 


; 351 " 


%1 


644 




! 803 


551 


746 


555 


6 

* 


814 


586 


785 , 


,517 



•Scores based on appropriate second-year posttest levels of the California Achievement Tests., Reading 
Vocabulary and Comprehension, and Mathematics Concepts and Computation, Intercorrelations are cor- 
rected for unreliability of subtests' fc * * . ^ 

••Cain scores are computed as. residuals, with priest predicting posttest of alte/nate-form sub-scale scores 
Intercorrelations are corrected for unreliability caused by pre-post time span, alternate forms, and subscafe 
„ unreliability, but not tor reduction in ln«»-<rore variance 1 
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variance This problem has been' called a paradox in the* literature, and is here to haunt 
us now.) ' , 

♦ *- 

Quite to the contrary, considering the general unreliability, of residual gam scores, the coef- 
ficients of collinearity reported above appear to be very large and are interpreted as arguing 
again for utilization of skill-area total scores. 

THE TESTING PLAN 

*•-*. , ' . V * 

On the basis 'of the information accumulatedjhrough the test screening procedure, the 

' visits to the test publishers, and other supporting analyses described above, the following 

test plan was developed. v /j 

FaJLoHfTe First Year Each student was to be tested with the 'grade-appropriate (at-level) 
test and o with the test one level below appropriate (below-level). Just Which level was 
at-level or below-level was largely to be determined by the grade-level recommendations of 
the publisher, but in some cases additional information had to be considered. On the 
assumption that each student would be acjmmistered two levels of the test, it was im^rtant i 
to assure that the lower level of the test would not be severely 'toppld out'* and trat the 
higher level of the test would not 'be severely 'bottomed out.'. Table 9-13 presents average 
item difficulties for all the levels and for grades Kindergarten through 6. - Avefagevjtem 
difficulties were extrapolated in a linear' manner for those grades and levels for which 
empirical data were not available. Inspection of the item "difficulties, and secondary 
inspection of,the percentile equivalent of 'chance' scores, resulted in decisions regarding 
which level wfculd be considered at-level for each grade and which would be considered 
below-level. Jhe tests in the heavy lines of Table 9-13 are the tests that werl selected. 

Based on the study of order effects, the testing plan shown in Table 9-14 was proposed for 
the 'fall of the first year. Yh^fall testing plan provided for the optimum ordering of test 
administrations so that the tests would not interact, and also provided days of non-testing, 
which the fiel^-test observers considered critical for obtairtinrg valid test results. 

Spring of the First Year. The testing plan proposed in Tabje 9-14 for the spring of the 
first year closely parallels that of the jail, with the exception that the levels of some of the 
standardized achievement test^ change to meet the growing skills of the students. 

After the fall testing: of the first year, the below-level and at-level test* scores for each grade 
level at each school were examined. On the basis of examination of means ^nd standard 
deviations, we^cbuld determine the appropriate levels for later years. 

Later Years. The testing plan for the study years after the first year is also presented irt 
Table 9-14. Notice that because there was no longer a need for double administration, of 
the standardized achievement test that the number of testing days has been reduced. . ' 
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Table 9-13 



Scale and Item Characteristics* (Empirical and Linearly Extrapolated) 
* of the Levels of the CTBS at Various Testing Grades and Times* V 



Level and - 
Grade Range 



Scales' 



No. of Items Min$ . 



Kindergarten 



Crack! 



Crade 2 



Grade 3 



Grade 4 * 



Grade S 



Grade 6 



Alternatives 




Fall 


Spring 


Fall 


Spring 


Fait 


y Spring 


Fa!! 


Spring 


Fall 


Spring 


Fall 


Spring 
/- 


Fall 


Spring 


* 49 

, 03 


39 


*57 
28 


74 
05 


91 

/OS" 


99 






















*03 


28 


. 50 
19 


65 
04 


80 
04 


95 








1 
1 


r 


r 


* 

\ 






43 ' 
03 


>9 




34 \ 
.47 


38 
41 


68 > 
07 


K 1 

98 






\ 












• 


44 

03 


45 




36 
47 


40 
42 


— 60 ' 
12 


99 




















04 








32 


46 
17 


59 
07 


73 

02 ' 




99 


* 




r 








\> 
41 

04 


t 7 




* 


29 


42 
18 


54 
11 


67 
03 


80, 


• 93 














85 
04 


46 










?3 

39, 


-~\4I 
* *4 , 


. 49 
16 


57 
. 08 


62 
° 6 * 


68 
04 


7'4 


80 


86 


t 


73 
04 


55 










30 
44 


42 

2S\ 


55 
13 


67 ** 

06 


73 
04 


79 

03 * 


85 


91 


' 97 




85 
04 


46 
















39 . 


45 
. 14* 

S 


50 
09 


55, 

°f 

53 
'05 


61 " 

.05 


64 
04 


68 . 

04 


73 
* 04 


55 






<*> 










40 


46- 
11 


52 
07 


64 
04 


66 

03" 


c * 69 
03 


85 
04 


46 








r 








f ' % 










'50 
12 


'53 
11 


\ 73 

1 04 


55 
















» 






i 




46 
16 


49 
13 



Level A 
KM 3 



Level B 
K6-1 9 



LeveKC 
1 6-2 9 



Level 1 
2 5-4 9 



A 



Level 2 
2 5-4 9 



Level 3 
2 5-4 9 



Reading 

« Main 
Reading 
Math 
Reading 
Math 
Reading 

* Math 
Reading 
Math \ 
Reading 
Math 



•First entry is average item difficulty index, second is«15e*rcentile equivalent of a chance score * — — 

•Levels enclosed in lines are those suggested for the S£S, based on considerations of samples, item difficulties, and scaling/equirements 



•Sub-scales are listed in Table 9-10 
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Table 9-14 
Testing Plan for the SES 



Day 



Grade 1 ' 



Grade 2 



Crade3 



Grade 4 



Grade 5 



Grade 6 



Fall of the First Year 



Week 1 
Monday & 
Tuesday 
Wednesday 
Thursday 
Friday 

>Week 2 

' Monday 
Tuesday 
Wednesday 
Thursday 
Friday 



/PT r 
CTBS-R-A . 
CTBS-M-A 

■» 

CTBS-R-B 
CTBS-M-.B 



PT/SAM PT7SAM 
CTBS-R-B CTBS-R-C 
CT6S-M-B -QTBS-M-C 



CTBS-R-C 
CTBS-M-C 



CTBS-R-1 
CTBS-M-1 



' SAM 

CTBS-R-1 

CTBS-M-1 

PAS 
CTBS-R-2 . 
CTBS-^-2 



SAM 

s 

CTBS-R-^ 
CTBS-M-1 

PAS 
CTBS-R-2 
CTBS-M>| . 



\ 

SAM 
CTBS-R-2 
.CTBS-R-2 

PAS 

¥ 

CTBS-R-3 
CTBS-M-3 



.Spring of the First Year 



Week 1 
Monday 
Tuesday 
Wednesday 
Thursday 
Friday - 

^Veek 2 
Monday 
Tuesday 
Wednesday 
Thursday 
Friday ' 4 



PT/SAM 
CTBS-R-B 
CTBS-M-B 

CTBS-R-C 
CTBS-M-C 



PT/SAM 
CTBS-R-C 



PT/SAM 



XTBS-R-C 



CTBS-M-C. CTBS-M-C 



r CTBS-R-1 
CTBS-M-1 



SAM^ 
SAM " 

CTBS-R-1 

CTBS-M-1 

PAS 



CTBS-R-1 * CT6S-R-2 



CTBS-M-1 



CTBS-M-2 



v SAM 
„SAM 

CTBS-R-1 

CTBS-M-1 

PAS * 
CTBS-R-2 
CTBS-M-2 



c SAM 
SAM 

CTBS-R-2 

CTBS-M-2 

PAS * 
CTBS-R-3 
Ct^S-M-3 



Week-1 

Monday J 
^Tuesday ' 

Wednesday 
• Thursday 

f riday 

Week 2 
Monday 



After the First Year 

PT//SAM) PT/SAjrt , PT/SAM , . SAM 
CTBS-R-X - « CTBS-R-X- CTBS-R-X ' < CTBS^-X 
CTBS-M-X CTBS-M-X CTBS&t-X CTBS-M-X 



v • 



SAM 
CTBS-R-X 



SAM 
CTBS-R-X 



L PT stanc 



PAS 



.CTBS-M-X , CTBS-M-X 



PAS * 4 



NotejFT stands tor Practice Test'; SAM Tor Student Affective Measure', CTBS for 'Comprehensive Tests of 
• Basic SkiHs.' followed by R for R*adingi or M for Mathematics' a*d by level designation, PAS for 'Prac- 
tical Achievement Scale' Parentheses around SAM indicate. that it is not administered during'the fall 
, testing, -but is during the spring The X indicates that ihe CTBS level is not known, but will be determined* 
a% a functionaMevel tor each grade at each school ■ " i 



ERIC 



■ 172 



.1 • 



'CHAPTER 10. THE FEASIBILITY OF USING CRITERION- * 
REFERENCED TESTS JN THE SUSTAINING EFFECTS STUDY 

Jacqueline KoseCoff and Arlene .Fink* 

The measurement needs of the Sustaining Effects Study (SES) and the theoreti- 
cal underpinnings and practical limitations of criterion-referenced testing systems 
were exhaustively compared in order to determine the feasibility of employing 
such an approach in a nationwide study. In general, the practical Jimitatidns of ' 
criterion-referenced tests and the level of effort needed to remedy qhose limita- 
tions led to the conclusion that the use of such instrumentation is not feasible, t ~ 

Criterion-referenced tests (CRTs) have become increasingly popular among educators and 
psyQhometricians. Perhaps the most important reason for their appearance and widespread 
acceptance can be traced to the new ways that had to be found to measure the effects of 
the educational reforms of the 1950-s and 1960's. Curing those decades, the conventional 
school curriculum was declared in need of reform, and a reassessment of the goals and 
objectives of American education*was made (Hofstadter; 1963, Davis and Diamond, 1974, 
Cronbach and $uppes, 1^69). .Innovative courses of study and instructional technologies 
**were ^subsequently developed, and, programmed learning and individualized instruction 
became commonly used teaching techniques. Ne\v ways, of assessing Student performance 
were needed that would correspond to the teaching innovations. 

Educators have v traditionally relied on paper-and-pencil achievement tests to measure 
learning, so it was naturaftor them to turn to test theoreticianUo provide them with alter-" 
native ways of interpreting performance on measures of education achievement for the 
new curricula and methods of instruction. The psychometrkians responded by pointing to 
two basic ways of assigning meaning to test scores. The first involved comparing the perfor- 
mance or behavior of one person or group with another person or group, |nd the second 
involved describing what a person or group can do or can lye expected fo do. Claser (1963) 
referred to these two ways .of, givuro rrteaning to test scores as norm -referenced and 
criterion-referenced, and recommended criterion-referenced score interpretationPfer the 
reformed curriculum and instruction. 

The reaction to CRTs was enthusiastic from fie start. Because they p'rovide score interpreta- 
tions in terms of the achievement of specific ^nd measurable skills and behaviors, .CRTs 
have had appeal to those directly responsible for the education of students and tine 
development and evaluation of educational programs. They also have had appeal to 
teafhers^who found the results of standardized tests inadequate to assist ^them in planning 
lessonsT and to many educators and psychologists whp^fudged standardized, norm- 
.referenced tests to be unfair and even biased against individuals from underprivileged and 
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minority groups., Finally, because the criterion-referenced approach was new, people saw 
it as ar\ opportunity to improve on some of the mistakes they perceived to be built into 
, * norm -referenced testing '* \ 

The popularity of CRTs and their sanction by both theoreticians and practitioners has led 'to 

their frequent use for instructional diagnosis and placement, and for measuring student 

achievement on ^ducational tasks or objectives/ In addition, .CRTs are now being suggested 

or used for other purposes, .such a§ the evaluation of educational programs and the 

National Assessment of Educational Progress (Wilson, 1974). In fact, rtijny state and federal 

agencies have specifically required evaluators to justify their selection 'of standardized 

rather than *CRT measures. 
■ - • f- 

To determine if CRTs are appropriate for the SES, we first examined the theory that struc- 
tures their development and validation to determine 'whether, on theoretical grounds 
_alone, CRT* are suitable or not suitable for national effectiveness evaluations.'The next step' 
was to fdentify a set of criteria for selecting test^ that are appropriate for such an evaluation; 
included within the set, of criteria was the stipulation that the test be able to provide scores 
amenable to CRT interpretation. We then feviewed currently available CRTs, using the 
criteria. Fiflally, based on the theoretical examination and the review, we formed conclu- 
sions concerning the use of CRTs in the^SES. This investigation is^prganized into four parts: 
the effectiveness evaluation qontext and the SES, a theoretical examination of .criterion- 
referenced testing; a review of currently available CRTs; and conclusions. 

THE EFFECTIVENESS EVALUATION CONTEXT AND THE SES 

The' evaluation of ap educational program mydlves the use of specific procedures 
that result in an appraisal of the program/s basic merit, and prpvides information about 
the nature and quality of th£ program's goals, outcomes, impact, and costs (Fink and 
Kosecoff, 1980). s ' 

Evaluation Contexts. There are two contexts in which evaluations of educational pro- 
grams are conducted. In one context, an evaluation is conducted to iprt^rove a pro- 
gram, and the evaluation's clients are typically the program's organizers and staff>lri the 
secood context,, an evaluation is conducted to measure, the effectiveness of a program, 
, an,d the evaluation'^ clients are typically the program's sponsors. The context for an evalua- 
tion is determined by the information needs of the individuals and agencies that must use 
the evaluation information. 

An evaluation is performed in an Improvement context wherj the evaluation's clients are 
concerned with finding out precisely where a change wojfd make the program better. 
Typically, the organi2ers of a still-developing program require jhis kmd of informati6n so 
* * that they can modify and improve, the program. On'4he other hand, an ^valuation is con- 
ducted in an effectiveness context when the evaluation's clients are particularly concerned 
•with determining the consistency and efficiency with which the program achieves desired 
results. Those individuals who sponsor a program's development, or who are interested in 
^ using the program" require this^kmd of information about a well established program's 
* outcomes and impact. • ' - 
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In an effectiveness evaluation, the evaluator usually assumes a more global and mdepen^ 
dent stance toward the program than in an improvement t context. In addition,' the 
effectiveness evaluator usually makes use of powerful, experimental design strategies that 
permit comparisons, rely on empirically-validated and standardized 'instruments, and 
•errtploy statistical and other analytic methods that allow inferences regarding the program's 
comparative value. , • 
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^Effectiveness evaluations of educational programs are Conducted, to appraise a program's 
overall impact and worth, and, if it is deserving, to certify it as being'able to produce certain 
outcomes efficiently. This type of evaluation isfrequently designed for programs that are 
relatively well-developed- and stable, having defined purposes and fixed proflfcs. Other 
names for evaluations of this sort include summative evaluation (Scnven, 1967) and 

' outcome evaluation (Klein et al., 1971). 

a » - 

The major focus.of the SES is the Title I program', originally funded through the Elementary 
and Secondary Education Act of 1 965: Title I represents one of the largest attempts by the 
federal government to assure equal educational opportunity for the disadvantaged. 
Because the purpose of the evaluation is to assess this established and long-lived program's 
impact m terms of costs and learning, it fits wTfhiri the framework of an effectiveness evalua- 
tion It is generally agreed (e.g., Alkin'et al., 1974) that information-collection strategies for 
projects like the SES should rely upon instruments that are known to provide relevant infor- 
mation and that have been demonsfrated to be valid arid reliable for the target population. 

A THEORETICAL EXAMINATION QF CRITERION-REFERENCED TESTING 

This section presents an investigation of the theoretical issues involved in the development 
and validation of CRTs. These issues include: defining CRTs; formulating and generating 
CRT objectives, items, and score interpretation schemes; establishing item and test quality; 
and examining the use of classical indexes of reliability and Validity. 

Definition: "A criterion-referenced test is one that 'is designed to provide a measure, of 
the extent to which educational purposes or tasks -have been achieved. All CRTs share 
several features in common: 
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• They are based on clearly defined educational tasks and purpose's. 

• Test items are specifically designed to measure the purposes and tasks. 

V Scores are" interpreted in terms of the attainment of preset criteria or levels of com- 
petence with respect to the purposes and tasks. 

All definitions of CRTs involve reporting test scores in terms of achievement, of educa- 
tional tasks. 

The difference between criterion-referenced tests and norm-referenced tests lies mainly in 
the metric used to describe their scores. Criterion-referenced tests report scores, using 



metrics such as mastery or percentage of an objective achieved, that are intendecTto 
measure levels of competence or achievement in terms' of a performance criterion.' Norm- 
referenced te^ts report scores, using metrics such as percentiles, and stanines, that are 
intended to pfecmiT "comparisons or rankings. All other differences between norm- 
referenced and criterion-referenced tests, sucb as the way each is developed and validated, 
are derived from the need to produce-tests that permit the appropriate score interpretation. 

Development of Criterion-Referenced Tests » 

Formulating and Generating Objectives. A clearly deftned set of educational tasks and- 
purposes (objectives) is required for CRT development. Such objectives are selected in at 
least six ways: . ' » « ; " 

1. Expert judgment. On the basis of knowledge and experience in th g^iej d, experts 
assess which educational tasks and purposes are the most important to measure. 

2. Consensus judgment. Various groups such as community representatives^curriculum 
( experts, teachers, and/or schooh administrators decide which educational tasks 

and purposes they consider to be the most important to measure (Klein, 1972; 
Wilson, 1973) ; , 

3. Curriculum analysis. A team of curriculum experts apalyzes a set of curriculum 
roaterials in order to identify, anct, where necessary, to infer the educational tasks and 
purposes that should be the focus of the test (Baker, 1972) , 

4. Expert analysis of subjects. An in-depth analysis is made of a subject area to identify a!! 
knowledge and skills that must be acquired if the area is to be learned w^ll (Glaser and 

. Nitko, 1971; Nitko, 1973). 

5. Theories of learning and instruction: The literature is reviewed and/or experts are 
consulted to formulate series of hierarchies of educational tasks and, purposes based 
uponr the results of psychological theory and research (Xeeslmg, 1975). 

6 Empirical studies. Experiments are conducted to identify objectives that are important 
* because the skills and knowledge represented are inherently essential. ' 

No matter how they are derived, educational tasks and purposes are usua|ly called objec- 
tives or behavioral objectives. However, it should be noted that these terms have a precise 
meaning to educators: ^ " 1 

* V * V. 

An objective is an intent communicated by a statement of what the learner is to 
be like v^hen he 'has successfully completed a learmng\experience (Mager, 1962). 

Developers of CRTs do not always use this defmiWi in itjypurest sense, father, they often 
use objective to refer to the 'content that is suppUsed to have been learned (e,g:, 
equivalent and nonequivalent sets in sixth-grade math) and sometimes also include the 



behaviors a student is supposed to exhibit (e.g., naming the first five Pr«*dents ; of the 
United States). ' w 

Another issue concerning educational tasks and purposes, e.g., objectives, relates to the 
rules needed for writing objectives and ,how broadly or narrowly they should be stated. 
Formal rules for generating and stating objectives aje needed {q ensure the uniformity, 
manageability, and comprehensiveness of the set' of objectives or domain that the 
r CRT measures.* 

Still another issue deals with how a domain is organized. The objectives for a sjngle'cJomam 
can be grouped by grade levels; they can be organized according to majpr content areas; 
and/or they can 'be arranged into a hierarchy according to the complexity of the behaviors 
involved or the order of instruction. 

Formulating and Generating Items. Once the objectives for the CRT have been chosen, 
the next step is to construct and/or select test items tp measure them. Thus is one of the most 
difficult steps in the .developmental process because of the vast number of/test items that 
might be constructed for any given objective, even wtfen objectives are relatively narrowly 
defined (Klein and Kosecoff, 1973). For example,. consider the following. objective: "The 
student can compute the correct product of two single-digit numerals greater than zero 
where the maximum value of this product does not exceed 30." The specificity of this 
objective is quite deceptive since there are 55 pairs of numerals that meet this requirement 
and at least ten different item types that might be used to assess^stud^nt performance. 

Further, each of the resulting 550 Combinations of pairs and item types could be modified 
in 'a variety of ways th^t might influence whether they have been answered correctly. Some 
of these modifications-are: * * 

1. Vary the sequence of numerals (e.g., 5 then 3, versus 3 then 5). 

2. Use-different item formats (e.g., muitiple-choice versus completion). 

3. Change the mode of presentation (e.g., vyntten versus oral). * 

4. Change the mode of response '(e.g-., written versus oral). J 

* * * * • * . 

It is evident that Vhighjx specific objective could have a potential item pool oi well over ♦ 
several thousand items (lively, J 970; Hively et al., 1p73; Bormuth, 1970). Several factors 4 
influence the gumber of items constructed for each objective, e'.g., the amountof testing 
time available and the cost of making an interpretation error (such as saying that a student 
has achieved mastery when he or she has not). For some objectives/. many items are need- 



The set of objectives thaKa CRT measures is sometimes called a domain' of universe' of content (Skager, 
1975, Cronbach, 1971) However, the term domain is used by qt hers to mean*the rules for generating test 



items to measure a* specific objective (Hively, at al , )9T3) 
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ed to obtain a stable estimate of a learner's performance, whereas for* other objectives 
fewer items will suffice. *' ' « r * . 

A related issue in the construction and generation of CRT items is the degree to which the : 
items ^should bs sampled with respect'to their difficulty and content coverage within an 
objective- It is a well known and frequently used principle 6f test construction that even / 
slight changes In an item can affect its difficulty. The extent to which the items within an* 
objective are sampled with respect to difficulty has a direct -bean ng on the interpretation of 
the scores obtained In other words, if only the most difficult test aems are used, the phrase 
'achievement of the objective' has a very different meaning tha'p if the test items are 
samplesitaken from the full range of difficulties. . . 

Another issue, the instructional dependence of a G^j/ refers to the extent to which.the CRT* 
is designed for use with a specific educational program (Baker, 1972, Skager v 1<$*3). CRTs 
with a greater degree of instructional dependence have objectives and test itemS that are 
^associated with a particular curriculum or set of educational materials and tecnhiques. 
CRTs with a smaller-degree of instructional dependence, on. the other hand, contain objec- 
tives and test items that are not necessarily associated with the specific skills or content of 
* an educational program. However, such CRTs btill may have been developed from several 
educational programs and, consequently, have objectives and items -that , reflect the 
emphasis of these programs Conversely, CRTs with no instructional dependence are based 
on, a domain of content and behaviors that are independent of any educational program^ 
and, therefore, can be used to compare several different educational programs. (The. 
development of this latter kind of CRT is, in fact, similar to that employed in the better 
standardized norm-referenced Jests.) " 

^ * 

Consideration, of the ^ various issues involved in item generation for CRTs has^produced 
different strategies fpr'gener^ting and constructing items, namely: 

• Panel of experts. A group of .measurement and curriculum experts decided which 

items to use based on their knowledge of an experience in the field (Zweie, 1973). 
* . p . - * • . ♦ 

• Content/process matrix. Basically a* variation of the cfassical test construction tech- 
nique, this.approach involves developing a matrix of contents and behaviors (or tasks) * 
to be assessed, Items are then systematically 'sampled within this matrixjand perhaps 
alorife a third continuum of item difficulty as well (Wilson, 1973). 

, • Systematic item generation: Basic tern forms or specifications lire developed for each 
objective that define the range of item difficulties, all the relevant Contents arid\ 
behaviors, and stimulus and response characteristics of it^ms that can\e used to \ 

• assess the objdctive'(Hively, 1970, HiveJ/et al., 1973^Cronbach, 1971, Skag^r, 1973; ' 
Ropham, 1975). ' -X 

Formulating Score Interpretation Schemes. The uniquely distinctive feature t>f a CRT 
its ability" to provide a means for describing what an individual or group can do, knaw, or 
feel without having to consider, the skill, .knowledge, or attitude of others. Consequently, 
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CRT scores are reported and interpreted in terms of the level of performance obtained with 
respect to the objective(s) or domain on which the CRT is based. This type of score report- 
ing is very different from that used for norm-referenced tests in which scores are reported in 
terms of the performance of other individuals or groups, 

» 

It should be noted that scores on <TRT tests need not be limited to just a CRT interpretation. 
Other interpretations can alio be provided to expand cipbn the CRT interpretation (Klein, 
1970; Cronbach, 1970, Ebel, 1972). An example of one way of combining criterion- and 
norm-referenced information is. This sch6ol had an average score of 5*out of 10 on the 
objective (a CRT Interpretation), which is one standard deviation below the national 
average of 7 out of 10 (a norm-referenced, interpretation). The idea of using both types of 
interpretations is not new and does not reduce the theoretical soundness of the score inter- 
pretation Combining* score interpretations is particularly useful for describing what a 
student can be expected to be able to do and how exceptional or typical the performance 
of the student is. ~ 

Some of the different scores that can be interpreted in a CRT sense are: 

• Actual score The number or percentage of items CQrrect on a given objective - 

• True score^ An individual's or group's true level of performance on an objective/ refer- 
ring to the portion of the total universe of items for an objective that an individual or 
group could answer correct^ . The true score is the number of items that an individual 
or group would pass if every possible item were tested. 

• Mastery of an objectne/The achievemenj of a preset criterion level of performance!* 
To be legitimate, the criterion level should be meaningful and preferably empirically 
justifiable. For example, a criterion level of 7 out of 10 items has meaning if systematic 
stud> has shown that those who reach this level can actually do something that others 
who have not reached it cannot do, or if baseline data show that the average student 
achieves this level. • ■ 1 - 

• Performance time. This is the time it takes, in class hours or calendar da^s, for a 
student to achieve a given performance level. 

• Level readiness. This score reflects the probability that the student ^ready to begin the 
next level of instruction (this may be based on both the number of items correct and 
the pattern of answers given to these items)*. " 

• Item difficulty. The difficulty of an item is measured byThe percentage of , students who 
pass the item. (This score is given most often when only one item is tested per objec- 
tive; for example, National Assessment of Educational Progress.) 

» » * 

• Total objectives Mastered. The number of objectives passed or mastered by an 
individual or group. 

* «. 

• Total individuals who passed. The number of individuals or groups who passed* or 
mastered each objective. 



Validation of Criterion-Referenced Tests. When construction of the objectives and test 
items is complete, the CRT must be analyzed and validated. This process can involve giving 
the test to students and studying their responses (response data), or relying on a view of the 
test by experts (judgmental data). There is much ambiguity a£out the procedures appro- 
priate for analyzing and validating CRTs. There are, nevertheless, several measures (or 
dimensions) used for item and quality testing that are relevant to CRT validation, and that 
have associated with them rey,iew procedures, data-collection strategies, experimental 
designs, and statistical indexes. 

Establishing Item Quality. Following are several commonly considered dimensions of 
item quality: * ' < * x 

• Item.-objective congruence. A test item is considered good if it measures or is 
congruent with the objective it is supposed to assess. Item-o'bjective congruence can 
be established by using judgmental data. 

• Equivalence {internal consistency within objectives), A test item is considered good if 
it behaves like other items measuring the same objective. The concept is 'similar to 
item objective congruence, but its proper use depends on response data. Equivalence 
is usually measured by computing the biserial correlation between the score on an 
iterrvand the total score on all items measuring that objective. * 

• Stability (over time). An item is considered good if examinee performance is consistent 
from one 'test period to the next in the absence of any special intervention (e.g., in- 
struction is an intervention that can change performance^ Stability involves response 
data and can be measured by using a phi coefficient that correlates scores on the 
item from two different occasions. 

• Sensitivity to instruction. An it,em is considered good if it is sensitive to instruction, that 
is, if there is a discrimination in responses to thejtem between those who have and 
those who have not benefited from instruction. This measure of item quality is usually 
computed for CRTs that are linked to particular educational program^ >^nd it requires 
response data. . 

• Cultural/sex bias. An item is" considered good if it leads to accurate inferences about 
the knowledge, skills, or other attributes of an individual or group. Bias can be as- 
sessed using either judgmental or response data. , 

<- l 

Establishing Test Quality. There are seven ^dimensions commonly used to express -the 
quality of a CRT: 

• Test-objective congruence. Synilarto item-objective congruence, test-objective con- 
gruence assesses the extent tt> which the total test or subtest measures the relevant 
objective. Test-objective congruence is usually determined by using judgmental data. 

• Equivalence (internal consistency). Test equivalence measures the >|i^me^neity of test 
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- * items for an objective, that is, how coherently the tfest items assess the particular 
objective. This can be measured by using split-half correlation, Kuder-Richardson 
'formulas,, or coefficient alpha. > • 

• General stability (test-retest, or alternate forms): A test is stable to the extent that 
examinee response* are consistent from one test period to another or. across alternate 
forms of a test in the absence of any intervention. - * 

• Quantitative stability (number of items per objective and number of objectives per 
domain): There are two levels at which this type of stability for a CRT can be estimated. 

* At the first tevef, ^determination is made^of {he*umb£r of items that shQuld be tested 
in order to obtain a stable store on an objective. At the second level, a determination 
is made of the nurpber of objectives that Should be tested in order to obtain a stable 

• estimate of performance orf'the' domain. Stability can be estimated with response data- 
0 using correlation techniques and/or Bayesian models (Novick and Lewis, 1974). 

• Sensitnitj? to instruction. Sensitivity to instructidn refers to a test's ability to discrimi- 
nate between those who have aod -those who have not benefited from, instruction. 

- This type of measure of test quality , is usually obtained for CRTs that are linked to a~ 
specific educational program. <lt can be measured using response da{a by comparing 
scores of those who have and those whp have nbt received instruction. 

• Cultural/sex bias. Bia% in measurement -occurs when characteristics ofMhe test, the 
testing process, or the interpretation of test' resdlts leard to inaccurate inferences about 
the. knowledge/skills, or other attributes of individuals or groups (Ahderson et al., 
1975). This can be measured by ANOVA or regression techniques using response 
data, or by expert review using judgmental data. 

* « N " 

• Criterion validity. Criterion validity establishes the meaningfulness of the criterion, in 
terms of wfrch CRT" scores are interpreted. Establishing criterion validity is either a 
one-step or a two-step process. The first step involves assessing the meaningfulness of 
the domain. That objectives have been selected and organized to be, in themseJves, 
educationally significant, and that test items have been systematically generated to 
cover the objectives. In Step 2, 0 criterion validity is established through empirical 
means, and involves determining whether examinees who perform well on the' test 
have really achieved -the educational objective,, * . 

Using Classical Reliability and Validity. There has been considerable debate over the 
appropriateness of classical, indexes of reliability and x validity for CRTs. Some^ psychometri- 
cians have argued that since CRT items are selected toTTTeasLirt the achievement of specific 
educational objectives ancj not to discriminate among students, scores on CRTs can lack 
variation. This could arise in the following situation: Before instruction, none <.ofsthe 4 
students have mastered th t e objectives, and they might all receive a scqre of zero on th£' 
criterion-referenced pretest, whereas after instruction, they might all receive* very high 
scores on ^^criterion-referenced *posttest. A l^ck of variation in student scores, it is 
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claimed would cause the traditional indexes of reliability and validity (that-are bas$d on 
variance) to be inappropriate (Popham and Husek, 1969), " 

Others have argued that when CRTs are adminstered to a heterogeneous sample! that 
represents degrees of competence and receives different instruction on the, objective, 
there will be sufficient variation in test performance, to apply the classical statistical, for- 
mulas (Klein, 1970, Harris, 1973). This latter stance is becoming the accepted view, and 
it is now held that the classical indexes can misestimated for CRTs using a heteroge- 
neous population. > - 

Theoretical Value of CRTs for Effectiveness Evaluation. Relying on the preceding theo- 
retical discussion of the development and validation of CRTs, it is possible to ask whether 
based on theoretical considerations alone, CRTs are appropriate to measure achievement 
for effectiveness evaulalions and, iTi particular, for the SES. An effectiveness evaluation 
requires instrument tlA^e reliable and valid, and that provide meaningful scores that can 
be used to make deo^K about educatitmaLpolicy. In theory, there is ah orderly set of . 
developmental and validation procedures which, if followed properjy, produces CRTs that 
are based on well defined' sets of objectives and that can provide meaningful and useful 
score interpretations. Thus, from a theoretical perspective* CRTs 1 * are appropriate' and ) 
desirable for measuring achievement in effectiveness evaluations, and in the SES. However, 
there are two important caveats that must be attached to thus conclusion. First, thece ar.e 
persons wh<5 simply reject the notion of criterion-referenced testing and, with it, the 
meaningfulness of any CRT score interpretations. If an evaluation is being commissioned by 
individuals who share this view, then CRTs should not be used since the resulting in- 
formation, although theoretically sound, is likely to be. ignored^ Second, as is the case with 
norm-referenced tests, not all CRTs provide the same type of score interpretation. Some^ 
CRTs are reported and interpreted in terms of the number of items passed per objective, 
and many educators and policymakers find this type of score interpretation, by itself, to be 
inadequate for most effectiveness evaluation purposes. However, a rejection of this type of 
score interpretation is not equivalent to' a rejection" of the notion of CRTs since it is just one, ' 
of s^Veral acceptable ways of interpreting CRT scores, *and since there is no reason why 
CRT scores cannqt be supplemented by comparative data. { 



A REVIEW OF CURRENTLY AVAILABLE CRTS * 

To determine if currently available (1975) CRTs are technically sound, and if they have 
been designed to be easily used for evaluation purposes, a set of required CRT criteria was 
identified. Copies of currently available CRTs were obtained from publishers and were 
evaluated- using 'the review criteria. Our intent was to determine the practical, 
appropriateness of using CRTs for the SES, based o,n the results of the review. 

Generating Review Criteria. To structure the review of CRTs, a set of criteria was 
generated. The criteria reflect the characteristics generally accepted as being necessary and. 
appropriate for an effectiveness evaluation study. To obtain the cnteria, we reviewed the 
literature, the unique needs of the SES, and criteria already developed and used for review- 
ing achievement tests. ■ 
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Obtaining 0?7b. A list of publishers of educational tests was compile^ using t<?st review 
books (Buros, |965, 1972; Hoepfner et al., 1970; Hoepfner et al., 1971; Hoepfner et ah, 
1974), personal contacts, and library sources. It should be noted that publishers on the list 
were not r\ecessarily known as marketers of CRTs because it was not always possible to 
predict in advance who published CRTs and'who did not, and because it was considered 
important to indude as many publishers as possible in the review. 

A letter was sent to each publisher requesting that he send the following information about 
any criterion-referenced math or reading tests available: 

• Detailed descriptions of the test battery at each grade , * : 

• Sample tests for reading and math at each grade - • 

* • Lists of objectives or domains for reading and math at each grade 

• Directions for administering and scoripg reading and math tests attach grade \ 

• All technical manuals, field test reports, expert reviews, or test-analysis information 

• Information about special features like scoring services or cassette-recording directions 
, •' Cost information " / 

• Name and title of person to be contacted for additional information 

Some publishers responded to the inquiry by stating that although they did have math 
and/or reading tepts available, they were not able to provide more than a brochure without 
a charge SDC decided not to purchase any materials. This decision was based on the 
expectation that the size and scope of the evaluation study would warrant an investment 
on the part of an interested publisher. Justification dfSDCs decision appeared to follow 
since the publishers who did ?end materials usually had tests that were_ responsive to SDC's 
needs as described in its letter. Other publishers opted to lend SDC materials and requested 
that they be returned at the conclusion of this investigation. Only the- 28 CRTs in reading 
and/or math that wejre accompanied by copies of the test(s) and test manuals were received 
and reviewed; the names of the publishers and^the test systems are presented in Table 10-1 . 
Eac*h of the 28 C&Ts was indepencjently reviewed twice using the set of criteria generated 
for this purpose, and discrepancies w£re resolved by both reviewers. Any remaining ques- 
tions, usually resulting fjom unclear or insufficient information, were followed up 'with 
phon^ calls to the publishers. 



Explanation of Review Criteria 4 

There are 18 criteria against which CRTs were reviewed. For this review, reading and 
language arts were considered to be one subject area, and matWl second subject area. All 
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Test Systems That Were Reviewed, by Their Publishers 



lame of Test System 



Publisher 

American Guidance Service 

Key Math (Diagnostic Arithmetic Test) 
Woodcock Reading Mastery Tests, Form A 

American Testing Company ] ^ 

Mathematics Inventory Tests V* 
Reading Inventory Probe I ) — 

CTB/McGraw-Hill 

Comprehensive Tests of Basic Skills, Form S <CT&S/S)-Mathematics 
• Comprehensive Tests of Basic Skills, Form S (CTBS/S)-Readmg 

Diagioshc Mathematics Inventory 

ORBr^ (Objectives-Referenced Bank of Items and Tests) 

Prescriptive Reading Inventory / 0 

/ 4* 

Educational and Industrial Testing Service 

Tests of Achievement in Basic Skills (TABS)-Mathematics 
Tests of Achievement in Basic Skills (TABS)-Reading 

Educational Development Corporation 

Individualized Criterion-Referenced Testing— Mathematics * 
Individualized Criterion-Referenced Testing— Reading 

Harcourt Brace Jovanovich, Inc. (The Psychological Corporation) 

Skills Monitoring System— Readjpg 

Stanford Mathematics Tests (1973) 
, Stanford Reading Tests (1973) 

«» 

Houghton-Mifflin . . . 

Individual Pupil Monitoring Systems— Mathematics 
Individual Pupil Monitoring Systems— Reading 

Instructional Objectives Exchange 
yObjectives-Based Test Set— Mathematics • % 

' Objectives-Based Test Set 1 — Reading 

NationaLEvaluation Systems * • s 

Comprehensive Achievement Monitoring (CAM) Maintenance Package-Mathematics 
Comprehensive Achievement Monitoring (CAM1 Maintenance Package-Reading 

Richard Zweig Associates, Inc. 1 
Fountain Valley Teacher Support System-Mathematics 
Fountain Valley'Teacher Support System— Reading ^ 

Scholastic Testing Service 

Mathematics— Analysis~of Skills 
Reading-Analysis of Skills 

Science Research Associates 

Mastery An Evaluation Tool, Mathematics ^ 
- Mastery. An Evaluation Tool, SOBAR, Reading ^f- j$ 
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subtests or tests of individual objectives at the same level were groped together^ ancf con- 
sidered as a single reading or math test. Following is an explanation of the review criteria. 

t 

. 1 Coverage of specific skills, A test must cover skills in reading (language arts) and/or . 
math since the federal mandates for the SES require measuring achievement in 
these areas. Examples of basic skills are reading Comprehension, spelling, arithme- 
tic, and telling time, as compared to tangential skills such as using the library or.per- 
forming computations with ar slide rule. 

2. Grade-level coverage. Because of the scope of the SES, forms of th.e tests must be,* 
available for grapies 1 through 9. ' 

V ' 

3, Overlap of objective's: Some or all of the test objectives must be measured at each 
grade level in order to permit comparisons across grade levels, onovertime in terms 
of common educational objectives or skiljs. For this criterion/ objectives or test* 
items at different grade levels need not be worded identtcafly, nor need a formal 
means of identifying them be provided. 

4 Sumber of test forms, Due to constraints related to test administration and the time r 
available for testing, there should be a limited number of test forms at each grade 
level. Just one test per grade level is preferred in order to avoid problems with 
reliability that can arise when several test forms are used. , 

, % 

5 Directions for test administration. A test should provide thorough and clear mstruc- 
> tions for both the examiner and examinee. Directions concerning distributing- tests, • 

demonstrating sample questions, and administering should be provided in a de- 
railed and easy-to-read form, 

6 Special equipment required. Test* administcation should not involve any special 
equipment (such as cassettes or visual aids) aside from pencils and scratch paper. 

7. Time for testing. A reading or math test should be designed to be .completed within 
a class period. This usually involves no more than a maxmimum of 40 to 60 minutes, 
since CRTs have to fit efficiently within the time constraints of the planned informa- 
tion-collection activities. * 

\ 

8 Croup testing A test must be designed for group administration. This criterion was 
considered to be essentiafgiven the logistics of the information-collection strategy 
for the study which involves training local coordinators to train teachers .who ad- 
v minister, tests in gr^up situations. 

9. Itegi-objective match. Each test item should be coded to an objective or to the 
education^ tasks and purposes the test claims to"measure. 

10 Objective coverage. There should be a sufficient number of items to measure each 
objective adequately. The number of items per objective should vary as a function 
of how broadly or narrowly an objective is stated and its levef of difficulty. 
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11 Objective scoring: <A'test must use an objective scoring procedure since there are 
• ho plans to train individuals to interpret subjective scores. 

12 Machine'scoring options: The test must be available in or adaptable. to a macbiwe- 
scorable form. / - 

r > • 

13 Scpre interpretation scheme: A test must erpploy a criterion-referenced score inter- 
pretation scheme. Tests jjsing CRT interpretations in addition to other typj?s of score 
interpretation schemes were also acceptable. » 

14 Curriculum dependence: A t£$t should not be based on the objectives of any partic- 
ular curriculum or educational t - N - „ - 

1 5 . Cosfs o(fesf per pupil: The costs of testing pupils must be affordable for a large-scale 

study. • . , * 

16 Formal field test: A test should provide documentation of field test activities. I*is 
preferable that the field test participants be nationally and geographically represen- 
tative, be a probability sample, and include sufficient numbers of minority and 

> disadvantaged students to estimate bias. , V 

17. Information on item Quality: Information should be provided based'either on judg- 
mental, or response data, about item stability, sensitivity to instruction sex/cultural 
bias, item-objective congruence, and equivalence. a 

18. Information on test quality: Information should be provided on test quality, based 
either on judgmental or response data, to4nclude information about internal consis- 
tency, test stability, test-objective congruence, sex/cultural bias', sensitivity to in-' 
struction, and criterion validity. x 

*» • 

Results of the Review * 

The resutts of the tests reviewed foMhis study are presented below. Because many of the 2a 
CRTs were iptended for classroom rather than certification evaluation purposes, the rev?ew 
tended to make some CRTs look less excellent than they would have if they had been 
reviewed from another perspective, 

1 . Coverage of specific skills: Of the 28 tests reviewed, 1 5 were designed to aesess only^ 
reading skills an<£ 13 were designed to as?ess only math. All 2Qf tests foiused on 
basic skills in reading and/or math, rather than on tangential skills, and thus met 
the criterion. 

4 

* 

2. Grade-level coverage: Nine tests were available for grades K through 9, and thus 
met the criterion. 

♦ 3. * Overlap or objectives: Twelve tests appeared to measure the same objectives at all 
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grade levels. Sixteen tests appeared to have some overlapping objectives which 
were measured at most, but not all, grade levels depending'on the appropriateness 4 
of the objective and its Jevel of specificity. It should be noted that to make common 
objectives, test publishers frequently., used broadly-stated objectives, or skill 
categories whichTthey thert translated into tasks and skills of varying compl&*tfy for 
different grade levels. / * 

Number of test forms. Some CRTs had o^ly one test form per grade l£vel and others 
had as many as 3-K Usually those CRTs that offered a limited number of test forms 
per grade level would include .several objectives on a single forcn, while, those 
featuring more test forms per grade level would assess one or only a few objectives 
, per form. Three tests did not set limits on the number of testslhat could be created- 
from their bank of objectives and items. , ~ * 

Directions for test admnis(ration^?fa of the tests rrfet the criterion by 

providing adequate f directions both totTTe^-examiner and examinee for test 
administration." • ' 

Special equipment required Twenty-six tests required no special equipment for test 
administration and, therefore, met tjie criterion. Two tests required the use p,f tape 
recorders"or cassettes, arid one test provided no equipment information. 

m Time for testing. Only two tests met this criterion. More tests*(24) left tir^sfor testing 
open, but the reviewers judged that from their length they wouldVequire more than 
one hour of testing time s One CRT had no information about the time needed 
for testing. * 1 

Croup testing. Twenty-/ive tests'could be administered to groups and, therefore, , 
met the criterion. Two t^sts were designed for individual administration onty, and 
dne did not provide information with regard to administration. 

Item-objective match. Twenty-six tests had each item coded to an object!^. 

Objective coverage. The items tested for each objective ranged from one to 150. (It 
sh6uld b^L noted that the CRT with 150 items per objective w^ based on a 
computerized item bank from which tests of'any length could be generated) Tests 
*with five or more items for each objective were judged to meet this criterion. 

Objecthef subjective scorrt^. Twenty-seven tests employed an objective scoring 
technique. . \ . 

Machine scoring option. Eighteen tests met the criterion for machine scoring. 

> 

Scor£ interpretation scheme. Twenty-seven tests met this criterioh by using some 
type of criterton-netaenced score 4 interpretation scheme. Overwhelmingly]^ he , 
scheme was expressed as an arbitrary master/non-mastery score, or the number of 



items Gorrect on^a given objective. Of these same 27 tests, seven also" employ 
norm-refere'riced interpretations., ? < ' j 

% » . w • * 

14. Curriculum dependence. Twenty-two -tests appeared to have total independence, 
from'a. particular curriculum or instructipnal* program. Six other tests also 2ppea7ed 
to be rather general and independent, although they claimed to be based, in vary- 
ing degrees, «on a review of what is currently being taught in today's sch&bls. 

15. Costs b£ tests per pupil. Based on a purchase of &II reading or math tests at the thirdj 
grade level, costs ranged* from about five cents per student to $6.31 per student:* 
Tests costing less than $1.00 per pupfTper administration were judgecj to meet the 
cost criterion. • • v * 

16. Formal field test. Eight tests.provicled.dofeumentation concerning field test activities. 
However, the information provided was remarkably sparse, with several except * 
tions. Those who did conduct fiel^j tests usually attempted fo get. s'ome sort of . 
geographic and national representation. . V . * - XJ ** 

Information on item quality. Eleven tests reported having conducted item-quality 
studies based on boytb response, data, and/or expert review. Of *these* attention 
typically was paid to item-pbjective congruence, item stability ^or-equivalence,. and 
sensitivity to instructions ■ * : 

1,8. Information on test quality. Thirteen test ^publishers reported having conclutted ^gst 
quality studies; based on response data and/or expert review. Of these, internal 
"consistency, .stability, test-ob^ctive congruence* seasitiVity^ to instruction, and 
criterion validity (Step 1) w>ere more frequently attended to. * ^ . . ' 

4 Table 10-2 summarizes the results of the review /or each test^Each reding or-math test is 
identified by a numerical code, the codes ire necessary because the publishers submitted 
• their materials voluntarily and did not formally consent to a published review. 



Practicality of CRTs for Effectiveness Evaluation V ' * 

Relying on the preceding discussion of the characteristics of currently avgfable'GRTs, it is 
possible to ask whetfcer, based on practical considerations aJone, v CKTs are appropriate for 
measuring achievement in the SES. The answer is *no', frormfhe reviews, if is clear that no 
CRT fully met the criteria Further, the review "not only uncovered 'serious practical 
problems th^t diminished the suitability of currently available. CRTs -for an effectiveness 
evaluation in general, but also brought to light additional problems which reduced th^ 
* usefulness of CRTs for the SE5 in particular. These problems are summarized below. 

^1 Many learning 1 objectives. Most of the CRTs reviewed had, a large number of very 
specific learning objectives that wer£ associated lessons. Thq, reason for the use of 
maqy, narrowly defined objectives can probably be traced.to the original use of CRTs* 
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Table 10-2 
Summary of Test Reviews 



Evaluation Criteria 



? 


1 . Coverage of 

Specific, Skills \ 


2. Grade Level " 
Coverage 


3. Overlap of Objectives 
Across Grade Levels 


4. Number of * i 
Test Items 


c 

8 2 
Q.| 

5 E 


, 1 

6. No Special Equipment 
for Administration 


7. Time for Testing 
Less Than 1 Hour 


8. Group 
Testing 


9. Item-Objective 
Match 


10. Objective 
Coverage 


I 11. Objective 
* Scoring 


12. Machine 
Scorable 


13. £core 

Interpretations 


14. Curriculum 
Independence 


15. Cost Per Pupil 
Less Than $1.00 


16. Formal 

Field Tesr- 


17. Information on 
Item. Quality 


18. 'Information on 
Test Quality 
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1 by teachers as an aide to individualizing and evaluating instruction. Nevertheless, an 
• effectiveness evaluation of just one year of instruction at one £rade level would 
generate information about an erlormous number of objectives, thus complicating the 
management; analysis, and reporting of data. 

2 Numerous test forms Many currently Pliable CRTs prov Je separate test forms for 
each grade level that measures just one or a few objectives. For example, of the 28 
tests reviewed, some had up to 31 separate forms for each grade level. The 
appearance of many test forms also probably reflects the original'mtention to use CRTs 
as classroom aides. In terms of national effectiveness evaluation, the logistics of 
administering a number- of distinct tests complicates information collection activities, 

„ increases the chances of making^ errors,- and* increases the costs of conducting 
the evaluation. fc 

3 Time required /or testing. Most available CRTs take m£re than an hour of class time, 
The/eviewer^found that 21of the 28 publishers claimed that their tests were untimed 
and thus left pacing^o the discretioaof t\\e examiner, however, based on th$ number 
of jest items>Jt is clear that one hour of test time is insufficient. In terms of the 
iffformation-collection schedule planned fbfthe SES, one class period of testing is the 
maximum time that can be devoted to a CRT. 

It should be noted that some, of the test publishers, recognizing time constraints, 
offered CRTs that had just one test item per objective. However, this is t(o\ a satisfac- 
tory solution since a reduction in the number of items will almost invariably bring with 
it a diminution in the test's ability to-measure with precision any of the objectives. • 

4 Matching CRT objectives with instruction: Using CRTs in effectiveness evaluations that 
involv^ more than one educational program means determining- relationships 
between otyectives pf the CRTs and the program so that achievement can be 
measured in. terms of the objectives emphasized in instruction. However, obtaining 
this information is costly and complicated. Teachers can be asked, for example, to rate 
the objectives of the CRTs jn terms of their relevance to classroom instruction, but 
teacher ratings can be unreliable? Instructional experts can be asTced to analyze 

' textbooks and curriculum guides; however, they cannot know for certain how these 
materials are being used in the cflassroom. - 

Another problem closely associated with that of relating CRT and instructional objec- 
tives concerns which objectives to test. Each student or classroom can be tested on 
. just those objectives that are derived from the curriculum being used; or they can be 
tested on a large number of objectives that encompass all curricula; or they can be 
.tested on a sample of objectives, some of which may, be relevant to the curriculum-, 
< while the others may not. Depending upon the^choice, the resulting evaluation infor- 
mation can be limiting in making comparisons or it can require considerable 
manipulation before interpretations can be made. * 

5. Identifying common objectives: A fifth problem with using CRTs in effectiveness 
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evaluation studies is that the same objectives are hot always measured at all grade* 
levels or if they are, there is no system for identifying common objectives, Although 
the sktfls and content associated with an objective generally become more complex 
with increasing grade levels, to make comparisons over time or across grades it is 
necessary to identify skills or objectives that are related in terms of a conceptual 
framework or general content area. For example, in the fourth grade,' a punctuation 
x objective might focus ont>egmning sentences with capital 'letters and ending them 
with periods, while in the ninth grade, a punctuation objective might focus on the 
pfoper use of semicolons as alternativesMo periods. Although both these objectives 
deal with ,the same skill area, grammar, unless they are formally referenced to that 
t general skill area, the evaluator must make this instructional type of decision, one that 
is ordinarily pot part of the evaluator's'area of expertise. 

6 Validating CRTs The procedures use*d'to validate CRTs are not very sqphisticated and 
field test results are not reported in any detail. When compared with the ' highly 

- structured field tests conducted for norm-referenced tests, most CRTs are deficient 
with/espect to the sample's size and representativeness, and/o> x the amount and preci- 
sion of data presented in technical reports. It must tie noted that test publishers have, 
probably been reluctant' to devote time and money to field testing because test 
theorists have not been able to provide them with an agreed-upon set of proceduV.es 
for analyzing and reporting the CRT field test data. Assigning blame, however, is not 
the issue since the fact remains that a paucity of da\a is provided-cpncerning the 
technical quality of tests and test items. . , 

7 CRT scores: Most CRTs produce scores in 6ne or two ways: either as the number of 
items Correctly answered for each objective, or sometimes as mastery or non-rjiastery 
scores, where 'mastery' means correctly answering an arbitrarily selected number of 
items per objective. These types of score interpretations are accepted by theorists as a 
legitimate way of expressing CRT test scores, and such scores may have meaning Tor 

* teachers who know^ their students and their curricula*. However, for purposes of 
effectiveness evaluation, these types of interpretation alonr^are inadequate because 
they provide Insufficient information for decision making and lose meaning outsiSe 
the classroom. * 

■0 

8 Financial considerations: A final practical problem with using currently available CRTs 
for effectiveness evaluation purposes is that most are costly. This probably reflects the 
effort it takes to define domains; to develop the special features offered by CRTs, such 

' as referencing the objectives to various school curricula," and to provide many short 
test forms so that they can be used efficiently for classroom instruction -purposes. 

CONCLUSIONS " ) 

There is no currently available CRT that is appropriate for use in the SE§. This conclusion is 
based on practical, not theoretical, considerations. One major reason for the inappropri- 
ateness^of available CRTs is that many of them have been designed for Classroom, not 
evaluation, purposes, consequently, they are characterized by numerous, narrowly defined 
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objectives, each measured on a separate test form. In the context of an effectiveness evalua- 
tion, these CRTs produce unwieldy amounts of information, require too much time fof^ 
testing, and create logistical problems for testadministration. 

A second ma|or practical failing of currently available CRTs is that field tests are either not 
documented or are performed inadequately. As a result, the reliability and validity of these 
CRTs is simply not known, and it is inappropriate to provide decision makers wjth informa- 
tion of unconfirmed quality A third major 1 failing of available CRTs is that the'score inter." 
pretations are not as meaningful as the study demands. Mbst are presented as^ numbers.' of * 
items passed, without criterion validity information or comparative data or supplements. 
Two additional practical failings include the costs of CRT^jnd the absence of mechaflisms 
for tracking the same skills or objectives across grade levels \ 

( * * . N * 

-A CRT that is feasible to use to measure achievement in an effectiveness evaluation such as 
the SES should be based on a limited set' of objectives that represent essential competencies 
. and basic skills, be proven reliable and valid, and be'able to provide scores and are mean- 
ingful and useful If the SES were to incorporate CRTs as pSrt of the effectiveness evaluation 
in spite of' the Conclusions reached in this report, the -following efforts' would have to be 
undertaken £>nor to the administration of the test for poroses of interpretation: 

•^rie goals and objectives of each classroom, curriculum, and instructional sequence of 
„ all participants in the study would have to be obtained. This report has briefly detailed 
.the many practical problems this would involve. 

• Some universal^ set of objectives would have to be judgmentally created. This set 
would then constitute the focus of the CRTs. « f 

• One CRT system, preferably one that meets a maximum of -the criteria defined 
previously, would have to be selected and, if possible, items sampled for each of the 
universal sets of objectives. 

• The new CRTs would have to be field tested, both with students and with curriculum 1 
experts, to confirm that they meet the criteria. 

• Score interpretations would have to.be developed*a«d validated both theoretically 
• _ and empirically. * 

In essence, what would be called for is a test development and validation program fart*, 
^exceeding any so far completed for CRTs, rivaling'those of the best standardized norm-f?t 
referenced tests. Even then, the cost-effectiveness of such an option would be highfy 
questionable.^ 
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CHAPTER 1 1 . THE MEASURE OF GROWTH IN 
PRACTICAL ACHIEVEMENT 



Mary L. Spencer, David W. feessemer, Nicolas Fedan, and Bobby R. Offutt* 

Upon completion of a review of theory and practice in the assessment of func- 
tional literacy and practical achievement, it was concluded that no existing 
instrument would be appropriate for the students in the Sustaining Effects Study 
(SES). Consequently, an ( instrument-development effort was undertaken that * 
resulted in a reliable instrument tapping functional or practical reading and 
calculating that proved to be well-received by students. The reception is due to 
th& illustrated item stems and to their relevance to the lives of young people in 
most walks of life in* the U.S. : The Practical Achievement Scale is to be 
administered in both the fall and spring to all students in the SES in grades 4 
through 6. ^ ^ 

Due' to increasing concern regarding the use of standardized achievement tests with disad- 
vantaged ahd minority students, it was felt that a measure of more lifelike, non-academic, 
or functional instances of literacy in children should be employed to provide an adjunct 
index of growth in reading and math for the SES. Social indexes of literacy have usually 
been based on years of schooling or on grade-equivalent scores derived from standardized 
reading achievement tests. Assessments of literacy based on such indexes h^ve proved 
^defective in twch major ways. First, they apparently have led to a serious underestimation of 
the problem of literacy in the United States, and second, they have failed to provide a 
meaningful description of the actual capabilities of the population in terms suitable as a 
basis for policy. The defects of such indexes have been summarized by Harman (1970), 
Bormuth (1973), and Nafziger et al., (1975). 

In recent years, an increasing emphasis has been placed on determining the actual abilities 
of adults performing practical reading tasks involved in real-lifejituations, particularly those 
having social and economic utility. Harris (1970, 1971) surveyed adult performance in 
filling out common ^plication forms and answering questions based on newspaper 
employment advertisements. The National Assessment of Educational Progress (1971) 
conducted a nationwide survey of adult performances on a number of reading tasks based 
on practical materials. The Army has devised measures of reading, listening, and computa- 



*The authors were affiliated with Pacific Consultants, Inc. of Berkeley, California, which was under subcon- 
tract to SDC for the development of a test of functional literacy. Several test authors, publishers, and agency 
sources of tests were most helpful in the preparation of the instrument. Their efforts to relay materials and 
information in an expeditious manner are appreciated, particularly in cases where materials' pending 
copyright were entrusted to the reviewers. Mention is especially due to Dr. Norvell Northcutt, Adult Per- 
formance Level Test, Dr. Kenneth Majer, Basic Skills Reading Mastery Tests, Dr. Marilyn Lichtman, 
Reading/Everyday Activities »n~Life, Ms. Jan Algozine, New York State Basic Competency Tests, Dr. Harold 
Wilson, National Assessment of Educational Progress, and the EsfCffological Corporation, Fundamental 
Achievement Series. 



ERIC ' 193 219 



tional capabilities required in specific military occupations based on job-related written 
materials (Sticht, et al., 1972). 

In the course of developing tests designed specifically to measure adult functional literacy, 
substantial progress has been made in conceptualizing functional literacy and jn the 
. methodology of assessment- The work conducted by Educational Testing Service (ETS) has 
established a solid- body -of data on the reading activities of American adults (Murphy, 
1973) A survey based on a national probability sample identified the materials commonly 
read during everyday general activities, the duration of reading, and the perceived impor- 
tance of the reading activity. Based on the survey, a large number of performance tasks 
were developed. Each task was classified by type of material and by type of socioeconomic 
benefit provided by the task. An advisory panel evaluated the importance of the tasks, and 
a national survey of adult performances was conducted on 170 selected tasks. The data 
from the survey were later used to estimate the economic value of functional literacy skills 
(Murphy, 1975). 

A domain-referenced test of functional literacy (R/EAL) was developed by Lich'tman (1974). 
Through logical analysis, she identified nine common classes of reading materials represen- 
tative of everyday life activities and selected one specific type of material within each class. 
A behavioral objective was written to cover the terminal and subordinate tasks in each 
hierarchy, using realistic facsimiles of actual sample materials. 

DEVELOPMENT PROCESS FOR THE PRACTICAL ACHIEVEMENT SCALE 

In service of the need to supplement indexes of reading and math achievement derived 
from standardized achievement tests, an extensive search and review of relevant literature 
was conducted. This process resulted in a dissatisfaction with the term 'functional literacy" 
because of- its connotative limitation to reading-written* words because real-life activities 
indude\vork with numbers and computations, the less restrictive term 'practical achieve- 
ment' was substituted. The review process also resulted in the production of criteria for the 
evaluation -and selection of tests of practical achievement for school children; the 
implementation of those cirteria on a set of candidate instruments; and the conclusion that 
a new test would have to be developed to meet the goals of the SES. 

Definition of Practical Achievement No definition of functional literacy has yet been 
widely accepted. Bornauth has suggested an all-inclusive definition in the following terms: 

In the broadest sense of the wbrd, literacy is the ability to exhibit all of the behav- 
iors a person needs in order to respond appropriately to all possible reading tasks. 

Nafziger et al. (1975), poinjt out that literacy, unlike reading, refers both tp basic reaching 
skills and to socially appropriate reading^behavior. They indicate that functional literacy 
implies reading for a purpose, and a purpose related in some way to social utility. The U.S. 
Department of Education has been quoted by Nafziger et al. (1975), as defining a literate 
person m*the following terms: 



194^*' 



...one who ha* acquired the essential knowledge and skills in reading, writing, 
and computation required for effective functioning rn socfety, and whose attain- 
ment makes it possible for him to develop new aptitudes and to participate ac- 
tively in the life of his times. " ^ 

None of these definitions offers very specific guidance tp.the operationalization of the func- 
tional literacy concept as a basis for development of an effective assessment device. A 
definitioa is offered below that lends itself to operationalization in terms of the wider 
dimensions of practical achievement. * , 



Practical achievement in school children is* the capability of performing in a consistently 
successful manner tho& reading and computation tasks that: 

> * 

* .are normally encountered iathe course of everyday life activities by a majority of 
children in non-school settings; * " 

I • are normally encountered repeatedly, or involve a substantial duration of activity; 



• involve commerce with particular types of materials commonly found! in the environ- 
ment of the child; 



lai duration or 
only found in 



• involve non-technical language and symbolic representation; 
*• require specific observable behavior in relation tp the material; 

• serve definable types of social function; and^ 



• are regarded as important by the child or established authority figures, or have 
demonstrable and non-trivial socioeconomic 'benefit's.- 

Lacking a survey of reading activities for children comparable to that conducted with adults 
by ETS, it was relatively difficult to fully operationalize this definition of practical achieve- 
ment. Nevertheless, progress toward this goal was accomplished through logical analysis, 
the advice of experts, and by capitalizing on certain aspects of previous work done with 
adults. Toward this goal, a panel of experts was convened to refine the definition and to 
provide guidance [n th^ selection or construction of the needed test. This panel was 
cqmposed of Dr. Marilyn Lichtman, Virginia Polytechnic Institute and State University; Dr. 
James* Sanders', Western Michigan University; Dr. Thomas Sticht, Human Resources 
^Research (Denization, Western Division; and Dr^ James Vasquez, Far West Regional 
Laboratory. The present definition and the decision'to broaden ou[ concern to that of prac- 
tical achievement were direct results of a most rewarding interaction with the panel. 

The firsrmajor point of the definition that was adopted is thaMasks should be "normally 
' encountered in the course of everyday life activities by a majority of children in non-school 
settings." This aspect was operationalized by seeking panel nominations for representative 
tasks and by informally surveying activities of a' diverse sample of children, with special 

• * * 

* 



emphasis upon disadvantaged and minority children.,, As a result of this step, a list of life 
activities is presented in Table 11-1. In connection with each area ofrlife, several significant 
entities are listed that^could serve as effective stimuli-arousing associations to a reading' or 
computation task. r \ 

A systematic classification of materials was developed, both as a stimulus to memory and as 
a refinement and explication of the third point in the definition, and is presented in Table 
1 1-2. The table has been based largely on the classification of materials used by ETS (Mur- 
phy, 1973) and Lichtman (1974). ! 

There is presently no basis for explicitly defining the extent of language and symbolic 
representations involved in chilren's practical-achievement. 'Clearly, a child should not be 
.expected to read language and handle concepts that are not yet incorporated into his 
spoken language competence, or that lie outside his normal realm of experience. One such 
restriction,' confining the test to non-technical language, has been included in the defini- 
tion. Language is Considered technical if it occurs largely within a narrow field of social 
activity, if it is usuafly known'primarily by persons directly engaged in that activity, and if 

Table 11-1 



Areas of Life Activity for a Test of Practical AchievenW 



\ 



1 Personal Maintenance 

a. Food N . 

b. Clothing 

c. Health 



2. Personal Relations 
a. Family 
b Friends 
c Relatives 
d Pets 

*3. Institutional Relations 

a. School 

b Church 
. c Club 

d Police and Fire 

4. Neighborhood Locations 
a. Home 
b Shopping 

c. Postal 

d. .School 

1 e. MedicaJ , 
i Recreation 
g. Library 



°4. Home Maintenance 

a. Appliances 

b. Yard and Gardeft 

c. Furnfture 

- d. Radio, TV, Stereo 

^6. Travel 
a. Bicycle 

c Car 

7 Leisure 

a. Sports 

b. Games 
c Toys 



J8. Communications Media 

a. Newspapers and Magazines 

b Comics 

c Television and Radio 

: d. Books 

e. Movies 

f. Mail 

s g Telephone* 
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Table 11-2 



Types of Materials for a Test of Practical Achievement 



Signs, Labels 




Schedules, Tables 




Maps, Diagrams 
/ 'Categorized Listings 

Directions, Instrudions 


- 

- 

m 


Advertisements, Announcements ' * 




• x Forms 




: ' Personal Communications 




instruments, Controls 




Technical Documents 




r < . ^ * Discourse, Narrative 



most children are unfamiliar with it In the area of computation, no advanced forms of com- 
putation or symbols that children have not had an opporunity to learn should be included. 

rr 

At this time, it is also difficult to propose a system of behavior categories that will provide a 
^alid delineat ion of the specific behaviors involved ip functional literacy tasks. To a large 
extent, the nature of the reading tasks seem implicit in, and inferrable from, the material 
associated with the tasks. For example, the typical behavior involved in using a telephone 
directory is describable in terms of a systematic search algorithm, given 'a specific entry 
name, proceeding .through subgoals defined by a sequence of alphabetic cues, and 
resulting in location and retrieval of a specific numeric cftdrlfnfffthe listings. The usual 
behaviors associated with" filling out forms include reading and comprehending headings or 
questions which identify requested information, retrieving personaf information from 
memory or available records, and writing the information inappropriate blanks or check- 
ing off appropriate alternatives. , 

A systematic task-skills analysis of a considerable number of such tasks is required to define 
a romprehensive set of component skills covering the domain of' practical reading tasks. 
Lacking $uch an analysis, an informal list of behavior categories is offered in Table 11-3, 
based on an -examination of the materials used by ETS in constructing test it^ms (Murphy, 
1973) and thote* listed by Lichtmarf (1974). The kinds of material most usually found 
associated with each behavior category are also listed in the table. 

A sbmewhat different tack can be taken in specifying the computational behaviors. Real- 
world -computational problems can be distinguished on the basis* of the computational 
operations, the numerical and measurement content, and the means of computation. This 
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Table 11-3 

Preliminary Classification System f6r Behavior Exhibited 
in Functional-Literacy Tasks 



Behavior 


Material 


j 

Systematic Search 


Categorical Listings, Advertisements 


Choice Discrimination 


Signs, Labels, Maps, Diagrams, Instruments, Controls 


Preference Selection 


Categorized Listings, Advertisements 


4 

Retrieval of Personal Data 


Forms 


Selection and Storage of Information 


Personal Communication, Discourse, Narrative 


Performing Sequence of Operations 


Directions, Instructions, Maps, Diagrams 


Contingency Identification 


Signs, Labels, Directions, Instructions, Technical Documents 


Accuracy Verification 


Technical Documents, Forms 


Comprehension of Informatipn 


Personal Communication, Maps, Diagrams , 



provides the basis for a facet design of computational categories shown in Table 11-4.. As 
shown, the child will perform computational tasks associated with real-world materials 
requirmg operations, based on signed numbers representing physical quantities, expressed 
in units of measurement by appropriate means. 

ETS identified eight categories of socioeconomic benefits as shown in Table 11-5. These 
seem to overlap to Some extent with the catagoriesof life activity presented in Table 1 1-1 



Table 11-4 

Facet Design for Computational Categories* 



Operations 


Signs 


Numbers 


Quantities 


Units 


Means 


Addition 


Positive 


Integers 


Length 


English 


• Mental 


Subtraction . 


Negative 


Fractions 


Area 


Metric 


Paper and 


' Multiplication 




Decimals 


Volume* / 


, Kitchen 


Pencil. 


Division 




Percentages 


Weighty-^ 












Time , 












Dry Measure 












Liquid Measure^ 






\ 






Speed 










* 


Money 












Cost/Unit 






1 it — ■ 






Angular 
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Table 11-5* 9 



| Functions of Literacy Behavior 



Economic ' » 




Occupational 




« Edutation/tulture 




Recreation* 




* * — Health 




Maintenance 




* Personal Relationships ' 




Citizenship 





Requirements Jor a Measure bf Practical Achievement. A general description of the 
characteristics desired in a test oPpractical achievements described. First, the instrument 
''must clearly measure the operational definition of practical achievement that was 
developed for the study. Accordingly/practical achievement is viewed as the reading and 
computational skills needed by children as they deal with the contemporary non-school- 
related wan^f. The test must be independent in the sense that it is specifically designed to 
measure practical achievement rather than being merely the reading or computational 
portion of.some achievement test battery. The level, range, and content of the test must, 
furthermore, b^ appropriate, for school children in grades 4 through 8 from disadvan- 
taged backgrounds. 

'» . , , 

Some of the more practical requirements ate that the cost of the test should be within the 
normal range of costs for comparable tests, it must be capable of being group-administered 
by non-expert school personnnel employing uniform procedures across the country, and it 
should be machine scorable. 

If the test is a norm-referenced test, the norms pertaining to the population of the study 
should be available. If the test is criterion-referenced, the criteria on which the test is . 
developed should relate to the study's definition of practical achievement and the criteria 
„ for score interpretation should be clearly specified. Technically, the test should exhibit 
adequate reliability and validity indexes. 
» 

Criteria for Selecting a Test - • 

/Characteristics of the test, the nature of the examinees, and the purpose of testing are im- 
portant factors in selecting the practical achievement test. The criteria for test selection are 
.based largely on the general guidelines provided by the American Psychological Associa- 
tion's Standards for Educational and Psychological Tests (1974), and the criteria employed 
in the C5E Elementary School Test Evaluations (Hoepfner et af , ,*'1976J. Additional criteria 
were suggested by review of reading and literacy tests for a Right-to-Read Evaluation, and 
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the examination of tests of adult functional literacy performed at the Northwest Regional 
Educational Laboratory (Nafziger et al., 1975). The criteria define factors relevant to test 
selection, but were not concerned specifically with the measurement of practical achieve- 
ment in grades 4 through 8 for the purpose of program evaluation. A number of general 
recommendations were not suitable in meeting the special requirerr^ents of the Title I study 
and were therefore modified as necessary. The proposed selection criteria are organized 
'according to the six general areas: (1) test background, (2) psychometric quality, (3) appro- o 
priateness, (4) norm staQdards, (5) administration, and (6) interpretation. r 

Test Background. The purpose oMhe test should 6e explicitly stated, and examples ' 
should be mentioned. The purpose statement should be clear to those individuals likely to * 
administer the test. The construction. of the test should follow closely the purpose for which 
the test was built. Thus, a diagnostic test should state how the test's purpose translates or 
agrees with the scope of tasks and operations to be covered. Such scope should be limfted, 
well defined, and detailed. The test should indicate whether differences among minority 
groups were considered during test construction. If such consideration were exercised, 
items would have been sampled that depict the actual behaviors of students in these groups 
in extracurricular life activities. y 

Psychometric Quality. Criteria addressed in .this section pertain to the validity, the 
reliability, the comparability of test' scores, and thp quality of normative standards. Factors 
contributing to the credibility of a test as measuring practical achievement are considered 
in terms of content, empirical, and construct validity. In terms of content validity, it is 
^desirable that the test be representative of a definable population of items and perfor- 
mances with specific reference to the domain of practical achievement. The test should be 
specifically designed as a test^ of 'practical achievement. Disagreement on the validity of 
content will surely arise if the test was originaHy designed for some other purpose and if no ' 
explicit basis exists for judging the relevance of items. The" stimulus materials should be 
representative of those commonly encountered in real-life reading and computational 
tasks. Confidence in ttie representativeness of materials would be increased if a population 
of such materials were defined, and if the composition of the~population were described in 
terms of types or characteristics of materials and formed part of the definition of practical 
achievement used as the basis of test development. The performance, required in the items 
should be as clos£ an approximation as possible of th4tasks and skills commonly required 
I in real-life reading apd computational performances of'children. 

Explicit classification and/or description of *f domain of behaviors is desirable as part of the 
i definition used as a basis foe test development. The language and other symbolic represen- 
. tations should be representative of the symbolic content commonly encountered in real-life 
reading and computational tasks. Specification of the symbolic content in linguistic and 
mathematical terms can further strengthen and clarify the definitiqn of practical achieve- 
ment beyond the. material and behavior specifications usually considered. Such specifica- 
tions could be particularly helpful jri-defming levels or ranges of competence in relation to 
the domains of materials and tasks. The materials and tasks should be representative of the 
socioeconomic functions commonly encountered in real-life reading and computational 
"* Tasks A classification or description of socioeconomic functions, and the benefits or values 
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of performance, should be part of the definition of practical achievement that is used as a 
basis of test development. Suth a classification or appraisal system would help insure that 
functionally significant, rather than trivial, performances are represented. The materials and 
tasks of the functional literacy test should not be' referenced^ .specific program objectives;, 
the test is intended to provide an objective criterion by means of which the effectiveness of 
various programs can be Judged in the area of functional literacy. Th£ definition of practical 
achievement should be operationalized by the specification of a set of criteria! task? 
referenced directly to the characteristics of materials, behavior, symbolic content, and 
functions employed. ' 

* * ■ 
In terms of empirical validity it is desirable that there be empirical evidence relating the test 
scores meaningfully to other variables. It is advantageoyfcthat the test has been correlated 
in previous studies with other measures taken at .the same tirn^. The number anet^uahty of 
studies, the num&r.of variables, and the diversity of variables all contribute to the 
evidence bearing on the- meaning of a given literacy score. It is also advantageous that the 
test be correlated with measures taken at some later tirpe. Studies should also relate the test 
t© important psychological, educational, or socioeconomic variables. All these empirical 
relationships should be interpretable in terms of prevailing educational,, psychological, and 
^ socioeconomic theory. The measure of practical achievement should /elate sensibly to 
* variables that reflect components of functional literacy and to variables that are indepen- 
dent of functional literacy. Finally, the test should be sensitive to the effects of appropriate 
independent vanablesso that there is some assurance that effects will be revealed in tfae 
present study as well. 

\ ~ 

Considerations of construct validity have to do with the theoretical basis\)f the practical 
achievement concept, They are of lesser importance in judging validity than are content 
and empirical criteria^ jiveythe practical concerns of 'this 6tudy. The conceptualization, 
development, and empirical validation of the test should be- grounded on relevant 
psychological, linguistic, and educational theory. Particularly .important in this respect is 
the availability of a task-skills analysis which would define the components of practical 
achievement, indicate hierarchical relations among components, and the relationship qf 
performance to basic cognitive information processing operations. Such a theoretical fourf-"' 
dation is useful in generating hypotheses and interpreting results. Such formulations would 
provide a basis for tying changes in, practical achievement to specific educational practices 
and would pcpvide a° basis for interpretations of findings concerning relevant socioeco- 
nomic variables, and the function and benefits of.literacy. • 

The selection should also consider tjje test's reliability. If alternative forms are available, 
they should be based on parajlel items with comparable item statistics. The forms should 
correlate 80 or above at every grade level in the fourth- to eighth-grade range. Test-retesl 
correlations should be .80 or above over brief time intervals, i.e., one month or* less. 
Reliability coefficients could be lower over longer intervals, particularly when instructional 
experiences have intervened, having a substantial effect on the level of functional-literacy* 
performance. High internal consistency is not a necessary criterion for the test, since a test 
whidi is highly homogeneous is not likely to represent the full diversity of tasks that should 
be sampled for pt^ical achievement. A statistic that allows an interpretation of the^ 



reliability o? each score is desirable. If the test discriminates at various* age Jevels,*the stan- 
dard error of measurement (or each level, would show^iow well this drfferentiation is 
accomplished^Fin&lly, the procedures used in item sampling should be clearly defined'and 
rephcable. It is necessary that test information indicate the relevance and represen- 
tativeness of the item pool in relation to the aspects specified in the definition of practical 
achievement Procedures used in selecting items from a pool for inclusion in the final test 
should be based on observations of actuaf behavior. Items should include a wide range of 
difficulties, including e spme itejns relatively easy for fourth-grade children, and some items 
relatively difficult for eighth graders. ' . 

! 4 

Appropriateness. The third set gf criteria concerns the appropriateness of the test Vor 
the examinees. They focus on the areas of instructiorfs, on items and format, and on pro-* 
cedure, and ensure that irrelevant sources of difficulty are^iminated from the test. 

©• i 

The instructions should be appropriate in orientation and tone, inoffensive in content, and 
comprehensive, with vocabulary and syntax suitable for children in the fourth- to eighth- 
grade range They should provide an honest explanation of the purpose # and intended use 
of the test Further they should precisely and completely describe all requirements of the 
tasks presented in the items so that the examinee has aH .the information heeded to adopt 
an effective performance strategy. Sample; items should be included that accurately 
illustrate task requirements and the level of difficulty of the tasks. A standardized script 
should be available for fluid oral feading by non-expert examiners.— 

The items shouTd be relevant, ijp to date, and interesting for children in the fourth- to. 
eighth-grade, range, so as to arouse intrinsic motivation in task performance without ex- 
tensive exhortations being required to irjduce cooperation, and effort. The content of the 
* items should not involve any invasion of privacy^ or any sexist, racist, or otherwise offen- 
sive aspects, y f ■ 

In terms of physical quality, the paper 'should be of good qtiality, the print bold and 
readable, and the illustrations clear, up to date, and preferably in color. The test should be 
effectively arranged and cued to facilitate recognition of items as units, the perception of 
the relation of item stems to a/iswers and examinee responses, and the progression of 
successive items and pages. Most examinees* should have, sufficient time to attempt most 
items. Sectioning of the test, with timing instructions for each. section, rrafy help to maintain 
appropriate pacing in the brief time allotted. The response* should be marked in a 'fashion 
permitting machine scoring. E^ch item should require one simple and direct /esponse, with 
4 . no multiple step$ or complications other than those intrinsic to the task represented by the 
item. Several items might be used, .based on the same stimulus materials, provided that the 
relationship ofceach item to the stimulus is clear. 

Norm Standards. Norm data are not essential in view of the large sample to be tested in 
the SES and the emphasis on program evaluation. It is desirable, however, that -some norm 
information be available for the fourth- to eighth-grade range that is representative of racial, 
ethnic,, sexual, geographic, and socioeconomic strata, ft is desirable that such data 
be reported separately as well as combined over the strata represented in the sample. It 
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would also be useful if item statistics were reported both for the whole sample and for the 
separate strat^. . 



Administration. Non-expert school personnel should be able to administer the test 
with very little 'training. The services of a specialist or a testing expert, or extensive training, 
should not be required. The test should require no more than 30 minutes of testing time. 
Tests taking longer than 30'rfrinutes -should be easily modifiable* for a shorter length, with no 
more than normally expected loss of reliability. Test administration in usual classroom 
settings, to group sizes in tf>e normal range for intact classroom groups, and without the 
necessity of special equipment, is very desirable. 

The test materials should be entirely 9f the paper-and-pencM test variety, with no special ' 
manipdlanda, slides, or other unusual components; the costs should be in the normal 
range of paper-and-pencil tests. 

Interpretation. A high quality test manual Shouldbe available, describing the test and 
how to interpret its scores. The test scores should be highly meaningfuJ and understand- 
able by a non-technical audience including the general public. It is desirable that* the 
-implications of given test scopes fpr educational practice or public policy be clear and 
relatively direct. 4 m * 

Evaluation of Existing Tests m 

Six instruments were selected for review according to the criteria. They were selected 
because they possessed some property or set of properties that placed them within the 
range of promising instruments for'th* study's purposes. It should be clear from the 6utset 
that the judgments mage of these tests relate only to the potential usability of the 
instruments in the SES, and the test evaluations should not be construed as either indict- 
ments or, recommendations of the instruments for adoption in other contexts. Summary 
evaluations of each test are provided below7/^ 

' . A. 

Adult Performance Level Test of Adult Functional Competency (A PL). Although tlie 
APL measures both reading artd computational skills, it is intended for use with an adult 
.population. In the reviewer's judgment, most APL tasks are rr\uch too difficult for children. 

• « • 

From a test-construction point of view, although^fforts were made to select tasks actually 
encountered by adults, >there was no evidence that observations of adult behavior were 4 
made so as to verify these tasks. The absence of validity data does not allow a judgment on 
the relationship between the APL and other tests and variables important to the concept of 
practical achievement. s V . 

Modification gf this test would depend on the availability of easier items. Since the APL was 
constructed for use with adults, it is unlikely that enough easy items can b£ obtained from 
the item pool to construct an instrument appropriate to the study "sample. _ V 

Basic Skills Reading ^astery Test (BSRMTl Two equally large problems exisf with the 

% ♦ *'*'.' 
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BSRMT from the standpoint of use for the StS. no Computational skills are included and the 
test wds constructed to be used with a population aged 12 to 18 years old. From a con- 
ceptual point of view there afe also problems, the most important of which is that the 
manner in which stimulus materials were obtained for item construction did not include 
actual observation of the people for whom it is intended. Modification of this test would 
hinge on the availability of an item pool from which easier items could be obtained. In 
addition, a computational subtest would have to be either constructed or adapted from 
some dfhfer source. 

Fuhdamental Achievement Series (FAS)* The FAS, as is, cannoUbe rated appropriate for 
the SES, primarily because it was intended for, tested, an£i normed with grades 6, 8, 10, 12 
and with various industrial groups. Additionally, there is no information on item construc- 
tion, and thus it is impossible to determine whether observations of reading behaviors were 
used for item generation. On the positive side, this test includes both computational and 
reading skills, and it has been widely normed. Modification of the FAS to suit the purposes 
of the study could probably £e accomplished, provided that the publisher has iter^ statistics 
on the remainder of the item pool. 

National Assessment^ df Educational Progress in Reading- Released Exercises (NAEPR). 
The advantages of the NAEPR exercises are that they have been used with children of the 
age group to be involved in the §ES and item sfatistics are available for several examinee 
varjables. Further, they could be formatted in a test package oTapp^opr^te length and to 
meet other criteria of administration. 

In spite of the positive qualities, the NAEPR exercises have several undesirable features. The 
most glaring rises from the lack of a computational subtest. In addition, the stimulus 
materiafs are not )udged to be intrinsically motivating and post-pilbt work w.ith them has 
revealed them to be sufficiently difficult for children In the fourth- to eighth-grade range as 
to generate some degree of examinee resistance. The item development process has the 
conceptual difficulty of haviQg been created by experts, rather than having flowed from 
observations of real-life experiences of the population of potential examinees. 

The -possible utility of the NA^PR e^rcises tjp the Title I study is two-fold. First, an examina- 
tion of the item statistics of/certain iteTrryjudged to have qualities of intrinsic motivation and 
suitability may result m/ne selection of particular items. Second, the NAEPR items may 
have heuristic value to/the development or modification of a practical achievement assess- 
ment tool. As they stand, however, the collection ^f items represents an unacceptable 
means of assessment for the SE$. At a minimum, this collection would have to be modified 
and supplemented extensively. 

New York Basic Competency Test in Reading (NYBCTR). The lack of relevance of the 
items of the NYBCJR to children having the relevant age and socioeconomic attributes 
presents a serious drawback for the test's consideration. Futher, the test does not contain a 
computational section. The issue of item suitability is so problematic that it renders the 
modification of this instrument impractical/ * i 
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Reading/Everyday Activities in Life (R/EAL). The main strength of the jJ/EAL is the real- 
life characteristics of its item formats. Beyond this, little can be said to recommend it for 
use There is no evidence that the items can be used with children in grades 4 to 8. No 
computational subtest is available. The' use of cassettes or tapes makes it unusable for 
present purposes, although the newly prepared instruction 'script could alleviate this 
drawback. A passing grade of 80 percent is the only normative data presented, and is not 
related to^other kinds of real-world performances. 



Implication! 




RecQmmendations From the Test Review 



As a result of.the literature review, and the test-evaluation activities, a series of conclusions 
and recommendations emerged. None of the reviewed tests satisfactorily meets all criteria. 
The inadequacies of the tests for purposes of the SES are b&sic. No test could be found that 
is appropriate to the Yourth- to eighth-grade age group. All six of the tests contain various 
materials that are commonly encountered in real-life reading, with test items constructed 
from these materials. However, none of the materials was obtained by actual observation 
of the behavior of children Two of the tests measured both reading and computational 
skills but cannot be used tor the Title I study without modification because they are too 
difficult One could probably be modified at less expense, since it already coverslhe higher 
end of the intended sample, i.e., grades 6 and 8. Instructions, however, are tape recorded, 
and would therefore require modification. Although there is no evidence that the test items 
in either were built around actually observed, real-life reading behaviors, the APL items are 
accurate facsimiles of literacy stimuli commonly encountered by adults. Various other 
important criteria were lacking in these tests, but since the most basic criteria were not met, 
it is a moot point fo discuss additional inadequacies. ^ ^ 

The remaining tests, which measured reading skills only, have various advantages and 
disadvantages. NAEPR had the most information on eacR item. Although individual^ items 
rather than an administration-ready test package are actually available, if the construction 
of a test is envisioned, then serious consideration should be given to some of the items 
presented. Item statistics are available by sex, race, geographic region, size and type of 
community, and age. 

The favorable aspect of the BSRMT is that it has a test form for children 12 years of age. The 
lack of realistic facsimiles as item materials depreciates its value to the disadvantaged 
population of children to be studied. Modification of. this test would require that a set of 
easier items be added for 9- fo 11 -year-old children and that the representativeness of 
stimulus materials be improvech^ Further, test instructions would have to be converted to an 
oral presentation mode. 

The R/EAt* has the important feature of presenting its items as actual photographs, or true- 
to-life drawings of the objects that contain the reading matter, this is an extremely desirable 
characteristic of item presentation, especially where minority groups are to be tested. 
Unfortunately, most of the items were constructed for &n intended population of high 
school graduates and older. Modification of the' R/EAL for use in the study would succeed 



only if certain items were selected which are judged to have relevance to fourth- to eighth- 
grade children, and then supplemented with other easy and appropriate tasks. The instruc- <s 
tion script and dther details of administration would require y mmor modifications in order 
to make them entirely compatible with the testing purposes of the SES. 

f r 

Natest^xists that fully meets the needs of the study. It wouW^be possible, however, to 
construct a new test from multiple sources. This procedure would entail the combination of 
a computational section with either a partially suitable test of reading competenty, or with 
a reading section constructed of items drawrTfrom several instruments. Jn either case, some 
set of easy items would have to be added to both the reading and computation portions of 
the assessment tool. These easy items could be either new creations or, more likely, 
modifications of items extant in spme of the six tests reviewed. Another version of this 
option would consist of constructing a test from all possible sources, sampling and building 
items Any test produced by these methods would then have to be pretested, and modified*/ 
at least once before it would be ready for a field test. This approach unfortunately ' 
precludes the advantages of generating literacy tasks from the actual .experiences of 
disadvantaged children. For this reason, it is concluded that a wholly, new test must 
be developed. 

/ / \ 

The development of a new test would entail sampling the actJal reading behaviors of 
children who match the age and 'demographic chara^tet^tics of those participating in the 
SES. A technically sound item-building phase wouJjeKhen be required, with careful pretest- 
ing of the final instrument. The advantage of this option is that the finaj instrument would be 
a high quality test that' would be suitable for a wide range of future applications. Moreover, 
it would be most responsive to concerns regarding the testing of disadvantaged children. 

THE PRACTICAL ACHIEVEMENT SCALE • 

Efforts in the development of a custom-tailored test of practical achievement were based 
upon the desirable features and aspects of the tests of functional literacy reviewed, but 
focused upon remedying those aspects that made them inappropriate for use in the SES. 

The measure of practical achievement in school children was prepared in a manner consis- 
tent .with th§ test administration requirements specified in the criteria for test selection. 
Administration criteria are as follows: 

> 

Personnel. Non-expert school personnel should be capable of administering the test with 
very Kttle training. The services of a specialist or a testing expert, or extensive training 
should not be required." * 

Scheduling. The test should requll^ no more than 40 minutes of testing time (preferably 2 P 
minutes) on one occasion of testing. Tests taking longer than 40 minutes should be easily 
modifiable for a shorter length, with no more th^n the normally expected loss of reliability. 

Setting. The test should be capable of administration in usual classroom settings, to 




grotfp sizes in the normal range -for intact classroom groups, and without the necessity of 
special equipment. 

Scoring Method. The test should'be scored in an objective manner by machine. Machine 
scoring should be highly fail-safe and reliable, without complex error checking routines to 
proof the results. 

Material. The test materials should be entirely of the paper-and-pencil test variety, apd 
require no special equipment or materials. 

Clinical Pretest c 

. .; . x 

In initial test construction, input was solicited from teachers, parents and other social .oj\ 
4 educational experts. This provided information on the construct validity of the scale? 
Judgments , of item difficulty were aided by the results of an educational reading project that 
used stimulus materials and items from the Practical Achievement Scale (PAS). This study 
was conducted by a University of Califonia graduate student, who made" available to us 
some of his preliminary results. The data showed that for the age group used in the study 
(12 to 15), mos&of theltems were too easy since the proportion of children answering most 
items corrects was nearly. 95 percent. These data suggested {hat the items might be too 
easy for the target popufation of the SES. Consequently, several easy items were -deleted 
and more difficult ones we/e added. 
* " -*_ 

A clinical pretext was p^rforrhed in 'which the instrument was administered to nine children 
in the requisite 9- t^3-yeah-old range. In the clinical pretest, verbal comments were 
sought on each iterrfof each^stimulus object. On the basis of these comments, items were 
deleted or rewanv^Ao^^/ance clarity and avoid irrelevant or misleading wording or 
wording that was too difficult. The quality of each itenr^was also tested by ensuring that it 
called unambiguously for the skills that it was intended* to tap rather th^an involving some 
mental 'trick.' nrj 

\ 

Based on all the information gathered or made avatfable.to us, a further item revision was 
carried out, yielding the field-tg$t version of 49 items and 23 stimulus materials*. The easy 
questions required less than a minute and the more difficult questions seldom required 
more than a minute. Approximately 30 to 40 minutes % are needed by most children to finish 
the entire set of exercises, with the test administrator reading the qyestions and optional 
responses aloud. ' * • 

The Field Test * 

The n£w vision of the PAS would clearly have to be field tested before being employed in « 
the SES. The field test would*provid<e jnfomrc'ation on the appropriateness and effectiveness 
of jtems, on time and pacing, and on student reaction to the test. Based upon consideration 
of the life experiences of the country's major minority-group children, 23 jtern bases with 
49 items Were selected .from a larger pool of potential item bases that had been developed 
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and pilot-tested by the Pacific Consultants test-development team. The item bases were 
all piGtorial stimuli that presented information to be use9 in the solution of five alternative 
item questions. 

The item bases and questions were thought to be appropriate and interesting to the disad- 
vantaged and minority students as well as others, and were expected to provide a fairly 
good range of item difficulties. so that a final test could be selected on the basis of field-test 
data that would be generally easy for most students. 

The field-test version of the Practical Achievement Scale was provided with simple instruc- 
tions that indicated to the students what was to be expected .of them and how they should 
go about answering the questions in the test. In addition, the "I 'don't know" response 
alternative was explained as a response of last resort. The instructions and. the item ques- 
tions with the alternatives were read to the examinees, so that the effects of academic-like 
reading achievement would be minimized. The field-test version of the test was printed to 
ap^pear very much like an optically-scanable test instrument, which is the format to be taken 
by the finalform of the test. In this manner, any problems with format would be uncovered 
during the field test. 

The 'Field-Test Coals and Plan^Jhe PAS was field tested ~at 15 schools across the coun- 
try that vve/e selected according to the four stratification types: (1) Northeastern, poor, 
urban; (2) Southeastern,, poor, non-urban; (3) Northwestern, non-poor, urban; and (4) 
Southwestern, non-poor, non-urban. Two classrooms at grades 4 and 6 at each of the 
schools were administered the test under the standardized field-test procedures. A total 
of 750 fourth-grade students and 758 sixth-grade students completed the instrument in a 
valid manner. 

The main purpose of the field test was to provide information for instrument revision. A 
second purpose was to provide some provisional statistical information about the final form 
of the scale. The questions that were posed for the field test are listed below: 

1. Which items (pages) can be eliminated to reduce the length of the test to about 12 to 
16 pciges? Three criteria were employed in the process of eliminating items: 

Are item difficulty indexes appropriate? Items too difficult for sixth graders or too 
simple for fourth graders would be nominees for deletion from the final form. 

» 

H6w closely is each item score (page score) related to the total score? Items and 
pages among those less correlated with ths total score would also be candidates 
for elimination. 

Do the item scores (page scores) discriminate between the fourth-grade and sixth- 
grade students? Items that fourth-grade students do better on than sixth-grade 
students^vould be considered for elimination, as well as items for which there is 
ncr notable difference between the two grades. 

• ' 234 ' 
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"2. How reliable is the totat score for the reduced test (after undesirable items have^een 
eliminated)? If the reliability is too low (less than .70), the least deletable items may be 
brought into the finat^ instrument to increase the reliability. 

3. Are the distractors of the remaining items functioning adequately? If some distractors 
(wrong alternatives) are so obviously wrong that too few students select them, or if too 
many students choose them, they will Be replaced with new alternatives (but the 
replacements will not be further field tested). , 

In addition to the analysis of response data from the field test, trained observers were 
present in the classrooms at most test administrations to make formal observations of 
problems. The observers were to obtain information on^the length df time it took to 
administer the test, problems that children had in following the instructions, problems with 
item pacing, problems with student interest or morale with items, and any overall impres- 
sions that might lead to the improvement of the test. 

Analysis of the Field-Test Data. The data collected from the field test were analyzed by 
computer and by manual tabulation of information not amenable to computerized analy- 
sis procedures. The analyses are presented below, in the same order as the goals of the 
field test. 

Are item difficulty indexes appropriate? Table 11-6, columns 2 and 3, present the percent- 
ages of stddents answering each item correctly, at both the fourth and" sixth grades. 
Columns 4 and 5 present the item biserial correlations with the total test score for grades 4 ' 
and 6, in response to the question-of how closely related each is to the total. A comparison 
of item difficulties between fourth and sixth grades can be made by comparing the difficulty 
indexes in columns 2 and 3 of Table 11-6. Based on the findings in Table 11-6, 19 items 
from 10 pages were deleted from the instrument to provide a test with 14 pages of 30 items. 
The retained items were then submitted to a program for the computation of the internal-^ 
consistency reliability of the reduced-length test. The reliability coefficient obtained is 
equivalent tq that obtained by the Kuder-Richardson-20 formula. The coefficient for fourth- 
grade students was .89, and that for the^sixth-grade students was .90. These estimates of test 
reliabjljty were considered satisfactory, so the item-selection procedure was not iterated to 
obtain a larger coefficient. The response frequencies to each of the items of the reduced- 
length test were then examined, and minor changes in four of the alternatives were made, 
these changes can_be expected to result in higher reliability for the instrument in late 
testing, and were made for this purpose. The reduced test total scores at the fourth grade 
correlated .84 with the standardized reading total (N - 315) and .69 (N - 331) with the 
standardized math total. At the sixth grade, the correlations were .81 (N - 307) and .74 
(N « 327), respectively. 

* 

Results of the Field Test Analysis. The reduced-length test had a total of 30 items on 14 
pages, as a result of the field^test analysis. Analysis of the observations of the test administra- 
tion resulted in the following recommendations, which were incorporated into the present 
form of the PAS and into the administrative procedures for the operational years of 
the study. 



Table 11-6 



Item Difficulties and Item-Total Correlations From the Field Test of 
the Practical Achievement Scale 



Percentage Correct Responses Total Test Biserial Correlations 

Cfade 4 Q»dc 6 Crade 4 Grade 6 

94 95 .41 - .68 

72 87 .59 .43 

A ' m X '- 63 \ - 74 

<• 11 47 -51 > .62 

86 _ 94 .70 76 

H! ■ 54 47 52 

86 92 _ .67 ^ 

86 .68 . 87 

77 11 66 -82 

• M % 71 . 66 

g 74 : 18 .35 

37 ' f 7 " 73 83 

37 61 . .81 • .75 

g 6 ' ' ; -M • . .61 

fl 11 68 77 

86 83 .68 .77 

J? .86 70 .78 

« • 53 .49 .43 

51 ' 88 .65 .72 



80 89 ; . < .70 

36 . \ 44 . .38 
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34 



61 85 .57 

41 63 " .74 

* 57 J - ^ 



.59 



89 83 ; .57 

\\ £ - 24 

67 > , 80 . .50 .57 

39 64 -83 .65 
£ 81 • .67 .77 

3 - 7 7 !j - 58 - 67 

53 7 4 _ .62 .70 

JJ 87 .65 .83 

u ' £ - - 7 ' 78 

34 56 . .66 . .62, 

.85 

. ' ;.35« 

43 .55 

48 - 68- .39 • . .49 

11 IS 67 77 

43 • 85 , .71 .67 

40 60 .45 .65 

t« • tl 30 • ' - 4 5 

38 58 .52 . .48 

.69 



.69 
.77 

' 3 o • 58 62 .62 

19 27 .24 3§ 

* ° . 82 • „ .56 ■ '6, 
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1. The easie$t;item was moved to the instruction page and incorporated into the test 
directions as an example. 

*» 

2. The test directions were printed on the cover page of the test. 

3. Items with difficult-to-read symbols or abbreviations were altered so that the test 
administrator would have no trouble reading aloud. 

4. Alternatives were not to be read to the students; only item stems were to be read. 
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CHAPTER 12. MEASURES OF AFFECTIVE GROWTH 

t ' f^lalph J. Melaragno^and Ralph Hoepfner 

Available measures of asp^ts' of affective growth in children were reviewed 
with the intent of selecting a s set of scales as outcome measures for the Sustaining 
Effects Study (SES). The Survey of Student Attitudes was selected to assess atti- 
tudes toward reading and math. Additional items were generated in a similar 
format to a&ess attitudes toward school in general. These three dimensions of 
attitude appeared in the field test to be relatively independently and reliably 
measured constructs. A primary versidn of the scales was administered in the fall 
and spuing to students in grades 2 and 3, but only in the spring to students^in 
grade 1. An intermediate form was administered in the fall and spring to students 
in grade 4 antf above. • J 

The primary thrust of all compensatory-education (CE) programs is to improve the cognitive 
^development of participating students. A parallel, but subordinate, objective is the im- 
provement of students' school-related, non-cognitive development. For example, the 
California guidelines for Title I state: 

Tfie main goal of every Title I project shall be to increase-the academic achieve- 
ment level of all eligible project participants to 'reflect a normal range and distri- 
bution of academic achievement for the target population as compared with the 
general population. Enhancing pupif self-image, motivating the pupil to achieve, 
improving his (sic) health, and raising his (sic) aspiration levels are to be consid- 
ered supportive objectives that must be attained in order to meet the project goal. 

Thus, while cognitive growth is the basic goal, non-cognitive improvement is viewed as an 
intervening variable— a precondition that enhances the likelihood of achieving cognitive 
growth, and that is Influenced in turn by the success or failure of achievement in the 
cognitive area.. f 

School personnel involved in compensatory programs typically express great interest in the 
area that is broadly termed 'attitude toward school' and that includes a student's feelings 
about peers, teachers, instruction, -and learning. Most CE program descriptions include 
a;concern with the improvement in students' likingfor different aspects of school work, 
particularly liking for activities in reading and math (which are the main areas of concern 
in CE). 

Students' affective behavior was to be measured as part oMhe SE5: The longitudinal nature 
of the study provides an opportunity to measure changes in affect that occur as students 
experience the cumulative effects of CE. We are concerned with measuring the extent to 
Which a student's affect is influenced by participation in a CE program," and .the differential 
effects on affect of programs with identifiably different characteristics. While direct assess- 
ment of affective development would be preferred (^.,,through>bbservation of students), 
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this is precluded by the size of the sample, the burden that would be imposed on students, 
and funding constraints. Thus, the measurement task must be accomplished less directly, 
through self-report instruments completed by students in the study. 



SELECTION OR DEVELOPMENT OF INSTRUMENTS 

I" 

There are two methods o( obtaining instruments for measuring affective behavior. One is to 
use existing instruments, and the other is to develop new instruments, either by creating 
new items or £y selecting items frorrV other scales. Each method has been used'in earlier 
, studies of CE and each method is accompanied by serious problems. 

In its evaluation of Follow Through, Stanford Research Institute (1970) used a variety of 
existing instruments. Generally, the results pointed. to limitations in the discriminating 
# power of tfie instruments and a lack of strong relationships with program characteristics, 
Poth the Equality of Educational Opportunity study (Coleman et al., 1966) and the Emer- 
gency School Aid Act study (Coulson et al., 1975) employed the technique of embedding 
affective items within a questionnaire completed by students. In both cases, the scales 
formed by the items were not successful in distinguishing between programs "with differ- 
ent characteristics. 

In a more narrowly focused study, the Educational Testing Service evaluated Compensatory 
fading Programs (ETS, ,1975) and, having found none of the available instruments 
- adequate to the study, developed inventories to measure students' attitudes toward 
reading. The ETS instrument yielded unclear results in the evaluation of the different 
reading programs. 

1 t 0 " 

' The rpajor problem with using existing instruments has been that they may not be appropri- 
ate to the needs of a given study. Typically such instruments were developed for particular 
respondents and specific purposes, and their utility, for other kinds of respondents is likely 
to be limited. The major problem in developing new'instruments for a particular evaluation 
effort is the time and expense involved in determining the reliability; validity, and othef 

psychometric properties of test items. A ' 

• * it* 
v 

Criteria for Selecting Affective Instruments. In investigating the feasibility of using exis- 
ting instruments or developing new ones for the SES, we first identified areas within the 
affective domain that appeared relevant to the study. We looked foY areas of concern to an 
evaluation of CE, areas that have been measured in previous large-scale evaluation studies, 
3fd areas that current research has shovyn to be related to students' experiences in school. 
Application of th^e criteria resulted in the selection of 'attitude toward school/ 'self- 
concept/ locus of control/ and 'achievement motivation' as the areas with the great- 
est relevance. 

Next, eight criteria for evaluating existing instruments were developed and applied to'aH 
instruments that had potential for use in the study. In an initial screening of possible 
instruments, standard references on t affective "measures and the files at the Center for 
the Study of Evaluation at UCb\ were examined, and the following criteria were applied to 
all instrument*: * 
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• Validjty and Reliability. There should be information in' a manual or in research 
literature indicating that the instrument has acceptable validity and reliability. 

• Interpretability Scores generated by the instrument should be easy to interpret for 
their underlying affective dimensions and should not require complicated or awkward 
interpretations. IJl 1 

• Age Appropriateness. The instrument should be valid "for some or a\\ of the ages of 
students in the study. 

• Administrative Ease. Classroom teachers should be able to administer the instrument 
after Iimited4raining. 

' • Scoring Ease. Because of the large number of students to be measured, the instrument 
should be designed for, or'lend itself to, machine scoring procedures. 

• Brief Testing Time. Because the amount of time available for measuring affective 
behavior is limited, an instrument should not require extensive time to administer. , 

• Minimal Response Bias. Younger students often demonstrate a bias toward socially 
desirable responses, and" instruments should be designed to minimize this bias. This 
can be accomplished by both the manner in which items are prepared and the types 
of responses called for. 

J • Commonality Across Grades. Because of the longitudinal nature of the study, either 
the same or highly related instruments should be used. In particular, instruments 
prepared in parallel to encompass different grade levels were preferred, 

9 . 

From an original set off over_60 instruments, 12 were judged to be of sufficient merit to 
warrant further consideration, with advice frOm'a panel of experts. These instruments are 
described here briefly. * - ) 

Animal Crac/rerslAd kins^and, Ballif, 1975). in its developmental form, known as Gumps 
Rookies. Measures achievement motivation. Developed for pre-school and primary-grade 
students. Requires 30-4^ minutes testing time. 

Attitude Toward Learning (Roshal et al., 1971). Measures general attitudes toward learn- 
ing at school, Developed at upper elementary level. Complicated response format.-Requires 
afxftjt 25 minutes. * » ' '~ < 

Attitude Toward Reading (ETS, 1975). Measures attitudes toward reading instru</iQn 
and reading-related activities. Forms for primary level and upper grade level. May-in< ucfc , 
fJositiye response bias, Requires 20-30 minutes. * 

Attitude Toward. School (Roshal et al.,^1971). Measures attitudes toward school in gen- 
eral. Developed at upper elementary level. Complicated response format. Requires about 
25 minutes. 



Children's Locus of Control Scale {Bialer, 1961). Measures generalized \oqus of control. 
Developed at all elementary-grade levels. Doubtful validity. Requires about 15 minutes. 

Children's Self-Concept Scale (Piers-Harris, 1969). Measures concerns children have 
about themselves. Developed for upper-grade students. Requires 15-20 minutes. 

Intellectual Achievement Responsibility Questionnaire (Crandall et al„ 1965). Measures 
control over, and Responsibility for, intellectual-academic success and failure. Developed 
for upper-grade stuqents; has* been used at primary grades in national studies. Requires 
15-20 minutes, w * ^ 

Locus of Control Scale for Children (Nowicki and Strickland, 1973), Measures general* 
ized expectancies for internal versus external control of reinforcement. Developed for 
middle and upper grades. Requires abouMS minutes. 

M-Scale (Williams, 19/2). Measures achievement motivation. Developed for upper- 
grade students. Doubtful validity. Requires about 10 minutes.. 

Quality of School Life (Epstein and McPartland, 1975). Measures satisfaction with school, 
commitment to classwork, reactions to teachers. Developed for -upper-grade students.* 
Requires about 20 minutes for full scale. 

Self-Concept of Ability (Brookover, 1967). Measures academic self- % concept. Developed 
at secondary-school level, modified for use at primary level. Requires about-20 minutes. 
r 

Self-Esteem Inventory (Coopersmith, 1967). Measures attitudes toward self in several 
domains. Devejoped for upper-grade stuJents, has been used in*primary grades in national 
studies. Requires about 10 minutes. 

/ 

The 12 instruments that survived the original screening are shown in Table 12-1; along 
with an indication pf how each instrument fared with the eight criteria described earlier. 
An X. indicates that the instrument was judged adequate on that criterion, '?' means 
that there was some doubt about the instrument for that criterion, while a blank indicates v 
serious concern. J 

Recommendations from Panel of Experts. A panel of experts on measuring affective 
behavior was convened to make recommendations. The panel consisted of: Dr. Joyce 
Epstein, Center for Social Organization of Schools, Johns Hopkins University; Dr. John 
Kitsuse, Department of Sociology, University of California, Santa Cruz; Dr. Melvin Seeman, 
Department of Sociology, University of California, Los Angeles; and Dr. James Vasquez, Far 
West Laboratory for Educational Research and Development. During the two-day meeting, 
the panel discussed issues related to, the measurement of affective behavior in the SES, 
examined instruments designed to assess affective&ehavior, and developed a set of recom- 
mendations. The major recommendations were: 

Use .availabl^fristruments rather than develop new ones. The panel felt that while 
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X indicates the.instrurnent was judged adequate on the criterion, T indicate^ there was Some doubt, 
and a blank indicates serious concern. ' ? , 



existing instruments all suffered from some shortcoming they were undoubtedly 
superior to any that could'be developed in the brief time pripr to data collection/ 
Specificafly, the 'panel recommended three instiuments: the Self-Esteem Inventory,^ 
Intellectual Achievement Responsibility Questionnaire, and the satisfaction-with- * 
school portion of the Quality of School Life inventory. Jn 'specifying these three in- 
struments, the panel recognized that they w&re choosing the best instruments among 
a less-than-ideal lot, and had particular concerns for validity and apprbpriatenessYdr <> 
younger students. The panel suggested potential modificaions^hat would be likely to . 
improve each instrument. v ' ' 
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2 Read all instruments to all students, in recognition of the obvious fact that the intent in 
measuring affective behavior is to assess those dimensions without contamination by 
reading ability W(^ile most instruments are intended to be read by the respondent if 
the cespondent has^^Csmpleled third grades the panel felt that many upper-grade 
students would lack suffib^ent skills to handle the reading tasks required; this can be 
overcome by having instruments read to all students regardless of grade level. 

3 Measure affective behavior befpre measuring cognitive behavior. In considering the 
total testing schedule for studeRts in the SES, the panel strongly urged that affective 
instruments be administered prior to cognitive instruments. The panel felt that stu- 
dents' attitudes would be strongly influenced by immediate occurrences, and that if 
the affective instruments were administered late in the week of testing, students might 
express negative views that would not be truly characteristic of them but rather would 
be in response to the (potentially frustrating) achjevement-test experiences. 

4 Measure students' senses of change. over time. The panel suggested that a valuable 
addition to the assessment of affective behavior would be the use of items that asked 
the extent to which the student was aware oi improvement in skills and changes in 
feelings and attitudes. In proposing this addition, the panel noted that the longitudinal 
nature of the study included the repeated measurement of students' affective behavior 
and thus allowed for the actual consideration of change over time; however, tfie panel 
suggested that a particularly useful piece of information would be the student's own 
avyareness^of that change. The, panel suggested the development of additional items 
tl^t would indicate the extent to which a student recognised that skills in reading and 
mathjiad improved, and that feelings abput self in, the school setting had altered with 
the passage of time and the experiencing of certain educational'activities. 



Survey of School Attitudes.. The most critical decision, based J6n the panel's concern 
with the adequacy of the* recommended instruments, was that the study should focus 
primarily-on the measurement of attitudes toward math and reading. Inasmuch as the prin- 
cipal objective of (IE programs is fo improve skills in reading and math, it was judged most 
appropriate for the SES to be especially concerned with students' attitudes toward those 
two curricular areas. Given this decision, a search was instituted te> jocate the«best existing 
instrument for attitudes toward reading and math. 

The search resulted in the selection of the Survey of School Attitudes (Harcourt Brace 
Jovanovich, 1975) for use in the study. This instrument had just recently been developed 
and released, and is ideally suited to the needs of the SES. The Survey of School Attitudes 
(SSA) is designed to measure student reactions tq reading and other language arts, math, 
science,, and social studies. Students indicate whether they like, dislike, or are neutral to 
different activities in each curricular area. The SSA can be used in group administration 
settings by a classroom teacher: There are two levels: Primary (grades 1-3) and Intetqiediate 
(grades 4-8). The two scales of interest in the SES are: * • ^ 

• Reading and other language arts: reading, working with words and sounds, writing, 
speaking; listening. 
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• Mathematics: concepts (of numeration, sets, etc.), computation, geometry and 
measurement, problem solving, charts and graphs. 

The SSA was standardized in 1973 on a sample of 13,500 students in grades 1 through 8. 
Twelve school systems in ten states participated in the standardization. The standardization 
sample was highly similar tojhe nation's population, as indicated by 1970 Census data, on 
the following dimensions: geographic region, socioeconomic variables, minority popula- 
tion, and community size. 

Item analyses ofjhp math and peading |gales, based upon combined grade samples, are 
reported in Tables 12-2 and 12-3. 

These data demonstrate that items correlate higher with their own scale than with the other 
scale, and that there ik sorne degree of commonality in the measurement of attitudes 
toward reading and- toward math. J 



Table 12-2 



Median Item-Scale Correlations From the Standardization Sample 





Test Form 

" 


Reading 


Math 






Form A, Reading 
Form A, Math 


54 

* .42 


.37 
.57 






Form 8, Reading 
Form 8, Math 


. ) 

.41 


.38 
* .57 
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Table 12-3 




A* 




Interscale (Reading and Math) Correlations 
From the Standardization Sample 








Level 


- Form A 


Form B 






Primary 
Intermediate 


.69 
.43 


.69 . 
.44 

V 





Reliabilities of the instrument were determfned both by coefficient a and by test-retest with 
alternate forms over a ten-day interval. Results are summarized in Tables 12-4 and 12-5^ 

These reliability estimates indicate that the SSA yields reasonably consistent and stable 
scores. 

Demonstration of $he validity of the instrument was approached from several directions. To 
find out whether the instrument actually measures student achievement, correlations 



y Table 12-4 

Internal-Consistency Reliability Coefficients 
From the Standardization Sample 





Test Form 


Of Coefficient 




Primary Reading, Form A 
Primary Reading, Form B 


81 

\83 




Primary Math, Fqrm A . 
Primary Math, Form B 


- - 85 
.85 ' 




Intermediate Reading, Form A 
Intermediate Reading, Form B 


.84 
.82 




Intermediate Math, Form A 
Intermediate Math, Form B 


r 92 

.90 




Table 12-5 
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Test-Retest Reliability Coefficients From the Standardization Sample 




Scale 


Test-Retest Coefficients 




Primary Reading # ^ 
Primary Math 


J .65 ' 
.65 




Intermediate Reading 
Intermediate Maih 


77 

,83® 



between SSA scores and achievement test scores were computed. They seldom exceeded 
30, leading to the conclusion that tfie SSA measures something different from achieve- 
ment, and is not overly influenced by achievement. To find out whether students respond 
in socially-desirable directions (to please the teacher), sub-scale intercorrelations were 
inspected to see af they exceeded scale reliabilities (they did not), and teachers were asked 
whether they felt students responded honestly (they thought they did). These results do not 
support the hypothesis of strong response bias. Finally, construct validity was approached 
through a series of factor analyses, which showed the SSA to measure separate atti- 
tude dimensions. 

ToJ*an$ider minority-group concerns, a substudy was conducted in which minority and 
non-minority students' scores were compared The two groups produced comparable 
results for both reading and math scales, and on both primary and intermediate level^. 



Each item of the SSA contains a picture of an activity related to a curncular area, and has an 
accompanying statement describing the picture. At the Primary level the statements are to 
be read by the examiner, at the Intermediate level the statements are also printed on the 
item The response options are three* faces, one smiling, one neutral, and one frowning. By 
marking the smiling face, the student indicates liking for the activity, marking the frowning 
faces indicates dislike for the activity, and the neutral face means the student is not sure 
how he or she feels about the activity. 

Modification* to the: SSA. Sprue modifications of the instrument were made for this 
study Since science and social studies are not relevant, those scales were eliminated. 
Scales for reading and math were extended by combining items from the alternate forms' of 
the SSA, resulting m two scales, reading and math, with 20 items each. The increased test 
length was deemed necessary to assess changes that take ptece longitudinally. The same 
instrument wifl be used for both pretest and posttest. This extension of the scales by com- 
bining forms is mentioned in the SSA Manual: v 

Users who require greater reliability for special applications of the Survey might ' 
consider administering both forms. . . . losing both fortes would increase 
reliabilities in accordance with the SpearmantBrown formula, e.g., approximate- - 
. ly from .80 to .90 or from .90 to .95. 

In addition, a few new items were added to the instrument to create scales that reflect the 
following attitudes toward school in general, self-concepts in the school setting, and 
students' sense of change in their own affective behaviors over time. These addit4enal items 
are described in detail later. The new items, along with the extension of the reading and 
f math subscales, make the length of the total instrument about the same as the original SSA 
with all four subscales. . >,* ~" • 



We decided that" the effective instrument used in'the SES would have two levels: 4 Primary 
version for grades 1 through 3 and an Intermediate version for grades 4 through 6. Each 
version would contain 56 items, as follows: 
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Reading/Language Arts* 
.Reading 

Working With Words 
Writing 
Speaking . 
Listening 

Other Related Activities 



Math . 

Concepts of Numeration, Sets, etc. 
Computation 

Geometry and Measurement 
Problem Solving 
Charts arid Graphs 

School-in-General t • 

Self-Concept in School 



ChangeNDver Time 

School-jn-General 
Self-Concept 
Mathematics 
Reading 



Number of Items 
Primary Intermediate 



- TOTAL 



7 

6 

1 

1 

2 

3' 
4 

,9 
6 
4 
1 

0 
4 
4 



2 
2 
2 
2 

56 



7' 
2 
2 
1 

"3 



9 \ 

4 

4 

2 

1 



C;2 

2 
2 
2 

56 



The 20 items for the scale on attitude toward math and the 20 items for the reading attitude 
scale were obtained by using all 15 items from Form A and five items from Form B. Items 
chosen frojn Form B were those that bore the least resemblance to items in»«i>rm A, to 
minimize the extent to which students will feel that they have already answered an item. 
When assembled* into a single instrument the reading and math items were alternated, 
somewhat reducing tfifc development of response sets by students. 

To measure attitude toward school-in-general, four items from the satisfaction-with-school 
scale of the 'Quality of School Life' inventory were used. The four items reflect attitudes 
that^are not tied to subject-matter areas, adults in school, or other students, but rather deal 
with school and class in a general manner. The items were modified to allow them to fit a 
response pattern of 'Yes,' 'Not Sure,' and 'No,' using the same three faces that appear in 
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the reading and math scales. Also, items cast in the negative were changed to the positive 
to eliminate difficulties students have in responding to negative items. The items are: 

I enjoy the work I do in class. 

School work is very interesting. 

I like school very much. 

J am happy when I am at school. 

• Self<onceptjp^school was assessed by four items selected from existing self-concept scales 
('Children s Self-Concept Scale' ^and 'Self-Concept Inventory') and modified to'use the 
same response pattern. The particular items chosen refer specifically to the student in the 
.school setting, rather than being related to self-concept in broader contexts such as home 
or play. The items are: * 

,1 can think up answers to questions. 

Hike to learn about new things. 

I^m good in my school work. 

I can learn things quickly. 

The eight items measuring the student's sense of cWange over time in affective behavior 
- were prepared following guidelines suggested by the panel. They, too, were written to use 
the .response mode described above/ The items are: 

I like reading more than I used to. 

* I do better work in reading than I used to. 

I like mathematics more than I used to. > 

y ' 

I do better work in mathematics than I used to. " . - 

* r 

School work is more interesting than it used to be.. • ^ 

I like school more than I used to. ' 9 

I am a better student than hjsed to be. " 
. I like my work more than I used to. _ 
A carefi/ examination of existing locus-of-control scales failed to uncover a reasonable set 
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of items that could be included in the affective instrument. The better scales (e.g., Intellec- 
tual Achievement Responsibility Questionnaire) are cast in the form of an qutcome for 
which the respondent is to choose one of two possible causes. Fonexample: "If something 
is easy to learn at school, it is because fca) you pay attention, (b) the teacher gives you lots of 
help." UsiQg items of this type would require a change in the instrument's response format, 
adding'to the time needed for administration. In addition, younger students,may experi- 
ence difficulty in responding accurately to sach items. On the other tend, Iocus-of-control 
iterqs cast in the 'yes-no' fowaaLig^., ''When I do good work in class it is because I am 
Mucky") are- pot the best ihdicatorsSSmternal-external locus of control. Locus-of-control 
items were, therefore, not included in thf new instrument. 

PRETEST AND FIELD TEST , 

When the new items described above were assembled into the second part of the affective 
mstrument, they w.ere randomized to reduce the development of a response set. To ascer- 
tain the extent to which Students of different ages have difficulties m understanding the new 
items or in dealing with a change in response pattern, and to e'Stimate the time required for 
administration, a nine-student clinical pretest was % conducted. The new items were 
presented after W\e items from' the attitude-to-reading scale and five items from the attitude- 
to-math scale, to provide a realistic trial of the hew items. Results. indicate that the items are 
understandable, ani^that students have little difficulty shifting responses. Time estimates 
were approximately 30 seconds per item, so that the full 56-item instrument could be 

admfnistered in less than 40 minutes. 

v. * 

v 

In March 1976,, v the complete affective instruments were tiredjxjt in a fieJd test under condi- 
tions similar to thos£ that were anticipated in the formal study. During the field test [\yf 
suggestions of the panel were followed, the affective instrument was administered on the 
first day of testing and examiners read the affective items to all students; The two levels of 
the Student Affectum Measure (SAM) were- tested in 15 elementary schools that represented 
a wide range of regional and, economic differences. The tests were administered as they 

- would be in the Operational years of .the study. " 

i' « , • ■ , s * - fi 

The Field-Test Plan ' 

Before the SAM was taken into the field for testing, a detailed plan was developed to guide 
. . the systematic analysis of the data so that each critical question would be adequately 
^addressed. The critical field-fest questiorfs were concerned with the issues of. instrument 
revision and instrument validation. 

Instrument tfgwvon: A -number of analyses were planned in the anticipation that the 
instrument might need revision. The specific questions are listed below: 

1 Are the items too complex, conceptually or linguistically, for elementary-school 
children' ls ? the testing procedure too difficult? These questions were to be answered 
on the basis of classroom observations of the fieJd testing by trained observers. A 
formal obsei^atforr .schedule was- developed and provided to the observers as a guide 
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to use in making observations. If the observations indicated that the students had diffi- 
culties understanding the items or the responding procedures, steps would betaken to 
reduce those problems through- revision of the instrument. " % 

,2. Do the items have satisfactory response dispersions? The amount of response bias 
• ^ (skewed response dispersion over the range of response alternatives) was expected to^ 
be Jarge, but there should.be some dispersion of responses over the three alternatives 
for each item. Any items vyithno dispersion or too great a response bias across gracie 
levels would be tagged for inspection and possiipfe revision*. 

Instrument Validation. Two other major concerns for the- SAM were the empirical in- 
dependence of the logical constructs (sub-scales) that are embedded m the measure, and 
the reliability of the measure and its potential sub-scale components. 

1. Should scale scores or one total score be used in the SES analyses? In essence, this 
question asks whether the logical constructs (sub-scales), described above in the sec- 
tion on modifications, have an empirical independence sufficient to warrant their 
separate consideration in analyses. The various components of the measure were 
incorporated because each was thought to be important in the assessment of affective 
growth. For this reason, it was most desirable that each of the components be 
considered for independent consideration in the analyses. On the other hand,, if there 
were no empirical support for the independence of the logical com'ponents, then 
separate analyses of them would result in the intractable problem of multiple collin- 
earity^of our dependent- measures. Two analytic approaches were employed to shed 
light on this question: 

a. If each of the new logical components (sub-scales) of the measure is separately 
scored, and the intercorrelations tjetween the components and the total score,' 
corrected for unreliability of each component,, exceeds or nearly equals the 
reliability of the component taken separately, only the total score r will be used in 
the study. This approach asks that the components have greater reliability than , 

; total scale coherence in order for them to be considered separately for ana- 
lytic purposes. 

b. If,empirical factors can be formed that loosely permit the conclusion that the 
components can be 'confirmed/ then the components will be consider||Lto * 
have jufficent empirical^support to justify their being analyzed separately fo^fe 
study. If no such permissive factor structure can be made to emerge,, then Such 
evidence will ^suppport the use of only, a. total secure for the affective measures. 
(This approach also allows for the weeding out of items that might not cohere in 
any of the possible empirical factors.) 



F f 

How reliabl&are the total scale and the sub-scale scores? The reliability estimates, in 
addition to pteymg a role in the first validation question, wijl indicate whether, or not 
additional items will be needed to obtain' an instrument of sufficient reliability to be 
employed in the SES, 
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The Field Test . 

t 

The SAM was tested at 15 schools across the country that were selected according to four 
stratification types: (1) Northwestern, poor, urban; (2) Southeastern, poor, non-urban; (3) 
Northvyestern, non-poor, urban; and (4) Southwestern/ non-poor, non-urban. Two 
classrooms each at grades 2, 4, and 6 at each of the schools w£re administered the test 
under standard field-test conditions. A total of 724 second-grade students, 743 fourth-grade 
students, and 753 sixth-grade students completed the instrument in a valid manner. 

Analysis of the Field-Test Data. The data collected from the field test wfere analyzed by 
computer and by manual tabulation and contenf-analysis of information not amenable to 
computerized analysis procedures. The analyses are presented below, in the same order as 
the goals and plans for the field test. « 

1 Are the items too complex, conceptually and linguistically, for elementary school 
children? Is the testing procedure too difficult? 

• Some evidence of children responding in a uniformly positive manner was 
observed, but it was not widespread or in response to confusion. 

% • Especially at the .higher grade levels, test administrators must pace.the items more 
rapidly in order to keep the students' attention. 

• The administration time ranged from 30 to 52 minutes for the second-grade 
students, from 16 to 34 minutes for the fourth-grade students, and from 17 to 30 
minutes for the ^ixth-grade students. 

• No general problems were observed with the directions to &e test. ' 

• Most items appeared to be understood, even^at the second-grade level. 

• The slightly differing response formats in the./ield-test version caused some prob- 
lems, but they will be remedied by one consistent response format and item 
format. * 



2. Do the items have satisfactory response dispersions? The response frequencies 
reported in Table. 12-6 indicate that it is not unreasonable to assume that there is a 
considerable response bias causing thje children to'mark the happy face, thus earning 
more positive attitude scores. - 

The possibility of response bias in the affective measure was not unexpected. The 
average dispersions, however, indicated that this bias might not be 'damaging to the 
e distribution of students over a reasonable range of attitudes. . 

3. Should scale scores or one Jotal score be used in the SES analyses? The first approach 
to answer this question was through factor analysis of the items at each grade level in 
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Table 12-6 



Average Response Dispersions for Three Grades' 



Stimulus 

v 


r 


Percentages of Responses 




VJldUC A 

(N « 724)' 


draae 4 
(N - 743) ' * 


Grade 6 
(N » 753) b 


Happy-face responses 


67 33 ■ 


62.82 


' 55.79 • 


• Neutral-face response^ ^ 


20.56 


24.84 


29.24 ■ 


Sad-face responses 


12 ' 98 


12.34 


14.66. 


Omitted responses 


0.87 , 


0.00 

<• 


0.31 



an effort to see if the separate attitudes presumed to be measured by the hypothesized 
sub-scales cpuld be said to have some sort of factorial existence. At each grade level, 
response data for the 56 items were submitted to a principal factor analysis. Item, com- 
monalities were Initially estimated by the multiple R 2 and then iterated to eight-factor 
convergence. In order to test the reasonableness of the factdr structure in a permissive 
manner, eight factors were txtracted and rotated from each, grade level correlatipfi 
matrix, regardless of, other extraction criteria. (If a factor-extraction cutoff had been 
established at an eigenvalue of ,1.00, 13 factors would have been retained for grade 2, 
* _ 12 for grade 4, and 1 1 for grade 6.) Jargeted.rotations to the eight hypothesized factors 
did not yield acceptably clear factor structures, nor were the factor scales very reliable 
or independent from one another. *" 

In order to obtain a clear factor solution withjnore reliable and independent factor 
scales, a four-factor solution was attempted on the data. The -hypotheses were that the 
four 'change in attitude' factors vyould be collapsed successfully upon their static 
analogs, •yeilding the following four factors: Attitude td Reading;* Altitude to Math, 

* Attitude to SchooJ-in-Gener^, and Self-Concept- in the Scbool Setting. } 

T ' 

* The fhrst four extracted factors for each grade level were then rotated to a target matrix 
composed of the four hypothesized factors. The factors emerged with, considerable 
clarity, but the internal consistency of* the last two factors was unacceptably low. 
Because of the low reliability, the last two factors were assumed to be collapsible, and 
three-factor solutions were attempted factor-analytically. The clarity of the three-factor 
solutionis depended to a considerable extent upon a ; catj:h-all'' bipolar fourth factor 
that served as a residual, but \h4 internal-consistency reliability of each of the three 
factors thus obtained was acceptable. The targeted factor-analytic solutions for three 
factors and residual for the three grade levels (based "on 724s§concl-grade students, 
743 fourth-grade students, and 753 sixth-grade studentsf are presented in Table 12-7, 
with hypothesized and targeted factor loadings presented in italics. The throe factors 
were named: ^ •* ' 




» Table 12-7 ' V 



Four-Factor Targeted Rotational Factor-Analytic Solutions 
for the Student Affective Measures at Three Grades 



- ■ — Tfcm 


^ Attitude to Reading 


Attitude to Math 


Attitude to SchooT 




Residual 


• 


" Cf.2 


Cr.4 < 


Cr.6 • * 


Cr.2 


Cr.4 


Cr.6 


Cr.2 


Cr.4 


Cr.6 


Cr.2 


Cr.4 


Cr.6 


f 

1 


.36 


*.34 




09 


13 


1 1 


01 


i 10 


18 


10 


01 


- 08 


2 


' " 31 


17 


25 


.36 


.42 


.ST 


13 


11 


24 


06 


14 


10 


3 


.40 


.42 


.40 


10 


07 


12 


41 00 


21 


17 


03 


.- 10 


- 11 


4 


06 


02 


22 


.47 


.3* 


50 


32 


19 


15' 


- 06 


1 1 


13 


5 


.46 


.42 


.43 


01 


2? 


> 25 


12 


J 2 


10 


- 07 


— Lrl 


1 0 


6 


19 


23 


27 


.44 


** .32 


.49 


09 


1*5 


14 


1 3 


— \J£ 


Ul > 


7 


.42 


.34 


36 


24 


- 01 * 


- 07 


15 


- 22 


30 ' 


OS 


- 10 


- 35 


8 


37 


- 08 


00 


.41 


% .58 


.65 


03 


31 


25 


1 2 


ni 

U i 


\ji 


9 


.51 


.42 


.36 


04 


33 


33 


06 " 


13 


16 


17 


01 


JXU 


10 


05 


14 


19 ' 


S7 


,.56 


'.59 


25 


18 


13 


- 02 


J0> 


03 


1 1 


.26 


.54 


.56 


26 


05 ' 


18 


24 


02 


- 06 


* - 09 


13 


18 




23 


01 


01 


.43 


.54 


.57 


- 01 


* 24 


26 


09 


07 ( 


- 07 


13 


.47 


.46 


.45 


19 


15 


12 


13 


08 


09, 




- 08 


- 07 


14 


18 


28 


23 


.44 


.34 


37 


> . 09 


05 


14 


- 01 




- 09 


15 


.38 


33 


.37 


' 19 


30 


24 


16 


16 


30 


- 01 


- 04 


01 


16 


30 


- 02 . 


13 


.57 


.52 


' .58 


1 1 


28 


23 


1 7 


02 


- 01 


*\? 


.51 


.41 


.44 ' 


28 


18 


32 


04 


16 


14 


12 


10 


10" 


18 


,35 


21 


11 


.52 


* .54 


.53 


03 


17 


18 




' 00 


qo - 


19 


.37 


32 


.33 


35 


02 


01* 


24 


10 


22 


- 06 


00, m 


-.15 


20 


27 


21 


22 


51 


.57 


.59 


12 


15 


15 


08 


"l 1- " 


°§ 


21 


.33 


.61 


.57 


24 


06 *. 


19 


07 


05 


- 02 


- 07 


* 16 


1 5 


22 


17 


15 „ 


16 


.44 


.56* 


'.64 


12 


16 


26 


- 01 


02 


03 


23 


.42 


.40 


.49 


25 


15 


05 


05 


M 


10 


1 5 




- uo 


^24 


20 


' 09 


10 


.58 


.66' 


.70 


21 


30 


21 


- 1 1 


on 




23 


s 34 


.30 


.44 


39 


04 


- 04 


15 


17 


26 


08 


- °5 


- 20 ~ 


26 


16 


- 32 


■ 1 • 


.51 


.47 


.52 


13 


09 


14 


04* 


pi 


- 10 


27 


.36 


.33 


• 41 ^> 


16 


34 


21 


24 


Jo 


29 


- 08 


01 


06 


28 


20 


31 


21 


.S3 


.50 


.50 


10 


t'06 




06 


- 03 


12 


29 


.27 


49 


.49 


21 


20 


29 


• 21 


19 


u 


- 16 


1 A 


j •) 


30 


14 


23 


21 


.40 


.53 


.54 


\i 


08 


>6- 


*oi 
yjj 


LAJ 




31 


.44 




.27 


05 


' 10 


or 


22 


17 


16 


18 


_ 1 1 


- ^ 7 


32 


f)& 


14 


20 


.30 


.60 


'.70 


M8 


19 


24 


- 07 


1 V 


* M 


33 


.40 


.51 


.50 


29 


28 


34 


20 


16 


17 


11 


09 


05 '* 


34 


04 


22 


14 


.62 


.55 


34 


• * 


11 


20 


- 14 


- 08 


- 12 


33 


.32' 


.27 


.32 


40 


- 02 


10 


21 


07 


13 


18 




- 26 


x> 


21 


17 


24 


.50 


.55 


.63 


05 


25 


21 


05 


. 20 


'© 06 


37, 


.46 




.43 


13 


• 46 


38 


14 


22 


19 


©8 t 


UV 


j ■) 


38 


11 


16 


20 


.65 


* .60 


.59 


27 


m 21 


19" 


- 07 


- 03 


04 


39 


.28 


.50 


.41 


31 


16 


36 


28 


' 24 


18* 


- U J 


i j 




40 


20 


21 


19 


.47 


.49 


.51 


.13 


05 


17 


- 04 


- 1 ' 


- 1 5 


41 




27 


29 


1 1 


10 


- 01 


.38 


.37 


.44 


- 06 


- '< 


- 21 


42 


J 21 


20 


22 


28' 


37 


41 


.47 


.53, 


.44 


25' 


22 


21 , 


43 


25 


18 


12 


- 03 


08 


15 


.46 


.41 


.3* 


- 17 


- 28 


- 27 


44 


13 


1 1 


22 


2> 


24 ^ 


^ 21 


.32 


35 


.26 


- 23 


, " »' 


- 26 


45 


1 1 


- 09 


- 09 


16 


26 


, 32 


.47 


.40 


.30 


- 27 


- 21 


- 15 


46 


20 


26 


v 34 


26 


33 


34 


' .46 


.40 


.47 


31 


28 


23 


47 


20 


28 


32 " 


23 


16 


28 


.30 


-.24 


.33 , 


12 


11 


- 04 


48 


13 


14 


16 


13 


18 


24 


.48 


.47 


.37 


- 27 


- - 31 


- 32 


49 


21 


2! 


18 




03 


09 


/ .43 


.34 


.33 


- 24 


- 18 


- 29 




,16 


22 


27 


16 


25 


22 


.60 


.60 


.64 


40 


35 


34 


51 


16 


f 18 


12 


18 


fe 


17 


.S7 


.50 


.57 


^24» 


19 


13 


32 


14 


12 


05 


13 


13 


17 


/ .42 . 


.47^ 


.37 


- 3* 


- 33 


- 33 


33 . 


02 


- 06 


-* 1 1 


32 


37 


49 


.54 


.54 


.40 


, - 07 


05 " 


10 


54 


14 


19 


27 


14 


21 


12 


.56 


.55 


.58 


* 31 


26 


20 


55 


10 


21 


19 


15 


15 


15 


63 


.66 


.70 


30 


30 


30 


56 


12~ 


26 


13 


20 


19*' 


28 


.62 


.63 


.66 


T 17 


22 


16 



Note Factor loadings »n bold face were targeted and were employed m the interpretation ot their respective factor* 
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Attitude to'Reading • 
, Attitude to Math 
Attitude to School 

i 

As a check upon the reasonableness of the factor-analytic solution, scores were 
obtained for each of the three independent scales and the scales were then intercorre- 
lated within grade levels over all students. As discussed above, these scale intercorre- 
lations should be sufficently small so that there is reasoh tq^elieve the scales provide 
assessments of different student attributes. Table 12-8 provicks the.scale interrela- 
tions for each grade, the internal-consistency (a) reliability coefficients, and the scale 
•>intercorrelations corrected for the unreliabilities of the scales. 

Table 12-8 * • . . 

c Factor-Scale Interconnections (Above Diagonal), Scale Internal-Consistency 
Reliability Estimates. (Diagonal in Italics), and Scale Intercorrelations 
Corrected for Unreliability (Below Diagonal) for Three Grade Levels 



* Factor Scale * 


Grade , 


Attitude to 
Reacftng 


Attitude to 
Math 


Attitude to 
School 




2 


>87 


.67 


.59 


Attitude to Reading 




.85 


•59 


* 58 * . - 


* • 


/ 6 

» 


.84 


.58 


.57 




2 


.75 


% .92 


56* 


Attitude to Math 


4 • , 


.66 


-.94 


,63 


f 


6 <1 


.65 


.95 


63 - 


Attitude, to School 


2 


.70 


.64 


V S2 


4 


- .69 


.71 


83 




6 


.69 


* ..71 0 


\J82 ■ 



As expected, the scale intercorrelations range in the .50s and .60s, which, when cor- 
rectedfor .unreliabilities, range in the .60§ anrf" .70s. When the fygh reliabilities of the 
scales (.80s and* .90s) afe, considered, it is concluded that each of the three stales has 
sufficient independence to warrant its independent consideration, especially when 
. employed as a dependent variable in the study of educational processes and services 
(But \{ is^recognized that the high inter-relatedness could cause analysis difficulties). 

Table* 12-<£ based on the facfor-ana lysis samples, fras been included e to provide infor- 
mation bearing upon the effects of the high item endorsement rates on the" distribution 
of scale scores. It is* clear that all the scales at all the levels are negatively^skewed, but 
in all cases the mean is more than one standard deviation from, the rpaximum score, ft 
is anticipated that the scales will provide a sufficient score range so that potential im- 
provements in attitudesxan be reflected with some sensitivity by increases in scores. 



Table 12-9 



Maximum Scores, Means/and Standard Deviations of Three 
Factor- v Scale Scores for Three Grade Levels 



Factor Scale 


Grade 


Maximum 
Score 


Mean 


Standard 




2 


60 


50.95 ' 


7.01 


Attitude to Heading 


4 


60 


50.82 


6.75 




6 


60 


49.04 


6.80 




2 


60 


49.55 


8.59 


Attitude to Math 


4 


60 


48.48 


8.80 




6 - 


60 


46 62 


9.47 


& 


2 


48 " 


41.93 


5.98 


Attitude to School 


4 


4a 


40 88 


6.04 




^ 6 


48 


39 35 


6.14 



4. How reliable are the total scale and the sub-scale scores? The internal-consistency 
reliability coefficient for each of the scores is reported in Tabje 12-8. Total-score 
reliabilities are .94 at all grade levels. • * v 

i 

Conclusions From the Field Test. The major conclusion drawn from' the field test and 
analyses was to retain the affective measure in a form very close *to its field-test form. Three 
scales can be scored and analyzed from the measure: Attitude to Reading^ Attitude to Math, 
and Attitude to School theJast scale being comprised of all the items newly created for 
theSES. - 
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CHAPTER 13-. THE EFFECTS OF OUT-OF-LEVEL SURVEY TESTING 



' Ralph HoeDfner and. Ming-mei Wang J 

two adjacent levels of the Comprehensive fpM of. Basic Skills were adminis- 
tered to each student in a natiofially representative' sample, the data %ere used 
to investigate problems associated with, out-of-level testing, floor and ceiling 
effects, and the identification of low achievers with different test levels. The floor 
and ceiling effects were found to be related to schootconcentrations of poor, and 
minority students. It is suggested that these school characteristics might be used 
' as a guide to select' better test levels' for a school, especially in survey testing 
■ where previous school-level achievement data are not available and where indf- 
vidualfy detefmjned -test levels wou[d be impractical. ' 

Out-of-leve.l survey testing has 'frequently "£g|| recommended as an appropriate method 
for assessing the ac bievement of students who'-are not functioning at the level&f their grade 
peers. The rationale is that using the grade-appropriate levelwith such students will pro- 
duee'only zero or chance-level scores and not truly reflest their (low) skills. To remedy this, 
several test publishers have made it possible to give«a1»one or more levels easier," but to 
reference scores to the studenfs' age-level peers. WhWemployed, jjU method usually 
involves the administration" and interpretation of achievement tests defined and intended 
for one or more grades lower.'thanthat of the children being assessed. The method rests ph 

* the assumption of the .unidimensional measurement off achievement growth* over the 

• gradi^and ignores or overlooks potential discontinuities in educational objectives oyer the 
grades. If out-of-levertesting is effective, it provides a solution to the problem „6f assessment 
of disadvantaged, minority, and handicapped students who are often not achieving at the 
level of their national-peers. 

' " <* 
One can irivqiediately foresee 'problems resulting from the implementation of out-of-level 
testing, however. Already, mentioned is the fact that out-of-l^vel assessment is very likely 
not to assess students on the learning objectives oHneir current instruction, but instead on 
the learning objectives of instruction of one or more years previous (but, see Plake and 
Hoover, 1979). An equally important problem stems from the fact that in the norming oper- 
ation a test level of a standardized achievement test is seldom administered to students 
whose assigned grade is far away (more than one level, removed) from that for which the 
level is intended. It is therefore necessary to rely on techniques of curve fitting and on 
seated scores. The interpretation of the normed scores must rest not only on the assump- 
tion^ (unidimensional) scalability pf the test levels, but also on that of the -functioning 
equivalence of children over a range of educational development. 

PREVIOUS STUPIES ON OUT-OF-LEVEL SURVEY TESTING 

These problems notwithstanding, out-of-level testing is still employed in. many situations; 
often enough so that several investigators have attempted to gauge the effects of the 
method. Ayrer and McNamara (1973) used the results of out-of-level testing in grades 4 to 8 



to study its effects. The level of test for each student was determined on the basis of" 
previous grade-level test performance pn a different test. The authors noted that among 
those taking out-of-level tests, the percentage of. students scoring below the chance level 
was much lower than what would be obtained had the grade-level test norms been applied 
to the scores on the out-of-level tests, with the grade-equivalent scorers an intermediary. * 
As the percentage of students in a school taking the out-^f-level .tests increased, average 
grade-equivalent scores and percentiles dropped off as much as three months and five 
points, respectively. This decrease was attributed to the increased floor available and the 
possible incomparability of normed scores among the test levels, at least for some groups in ~ 
the extreme ends of the norm distribution. 



Yoshida (1976) studied the effects of out-of-level testing with mainstream special-education 
students by having the teachers select the test levels to be administered on the basis of their 
inspections of the tests and their knowledge of the students. The out-of-level approach 
resulted in no reduction of test reliability indexes compared to those of the standardiza- 
tion sample, a large proportion of students scoring above the chance level, acceptable - 
ranges of 'item difficulties, positive point-biserial item-total correlations, and no 'apparent 
ceiling effects. . /< 

Employing teacher-selected test levels, Barker and Pelavin (1975) reported that of the 
students tested two or more levels below the publisher's recommended -level, 8.6 percent 
scored at or above the 80th percentile and 4.2 percent scored at or below the 20th percen- 
tile They contended Jthat the interpretable range of scores falls^between the 20fh and 80th 
percentiles. Presumably with the rationale that tests become unreliable in the extreme ends 
of the score distribution, they concluded that the fact that 87.2 percent of the students 
scored in the interpretable rang^as compared with the expected 60 percent in the norm 
group, indicated a success for their out-of-level testing in terms of teacher level selections. It 
is unclear, however, which grade norms were employed to determine the students' per- 
centiles corresponding to obtained raw scores in the selected test levels. It seems unlikely 
that norms for the students' nominal grades were used. This ambiguity notwithstanding, 
the agthors seem justified in their confidence that many of the 87,2 percent would have 
achieved scores so low, if they had been administered grade-level tests, as to vitiate their 
interpretation. They estimated that with the standard they set, over 55 percent of the stu- 
'dents would very likely Have earned uninterpretably iow scores if grade-level v tests h^d 
been employed. 

The extent of out-of-level testing is quite Substantial when teachers are allowed 4o select 
levels for each individual student based on. known achievement levels. Barker and Pelavin 
report that test levels below^fcie publisher's recommended level were selected by teachers 
for 64.8 percent of the students in grades 1 through 6. Of all the students, 26.6 percent were 
in fact administered^ests two or more levels below iheir grade^appropriate level. Yoshida's 
study (1976) with a group of speciaf-education students reported a disparity, as great as ten 
grades between the student's age-ptaced grade and the grade recommended for the out-of- 
level test selected by the teacher. The investigators of both of these studies felt that the 
teachers' judgments of appropriate test levels for their students were accurate. If nojhing 
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else, the use of teacher-selected test levels appears to be effective in alleviating many diffi- 
culties associated with the administration of test levels incommen&rate with an examinee's 
performance level, such as teacher's discontent, student's frustration, and c above all, score 
interpretability. Using two samples of low-achieving students only, Long, Schaffran, and 
Kellogg (1977) administered the grade-level and instructional level of a comrfronly used 
reading achievement test at the fall and spring of the school year. Comparing the grade- 
equivalent scores corresponding to group mean raw scores in the publisher's norm conver- 
sion tables, they found that grade-level testing yielded significantly lower grade equivalents 
than. instructional-level testing at grades 2 and 3, but at grade 4 the reverse was found. This 
finding was attributed to the inadequacy of the cross-level scaling o^jjie test. 

It should be noted, however, that many other factors would have contributed to this phe- 
nomenon. Converting mean raw scores to grade-equivalent norm scores is meaningful only 
if the grade equivalents are a linear transformatfon of the raw scores. Inspection of the 
norm tables shows that this is clearly not the'case, particularly at th<? ends of the score 
distribution. The sttidy sample^included only those expected to score in the lower end of 
the grade-level score distribution and this may reduce the grade-level test reliability for the 
group A third factor could be that grade-equivalent scores are typically extended to a 
lower range at the low test levels and thus provide a finer scale for the lower raw scores. 
The ctiange of signs in the discrepancy between the grade and the instructional-level grade 
equivalents at grade 4 may be explained by the increased floor effect that is, npt found at 
grades 2 and 3. 

Long, Schaffran, and Kellogg also found that at grades 2 and 3 more students in the sample 
were classified as low achievers wfth the grade-level teste and at grade 4 more were 
classified as low achievers with the instructional-level test. ft^lWd be pointed out that the 
study explored only one side of the classification problem sinceNonly low achievers were 
studied" It is this side of the problem where the difference betweeiVgrade-level and out-pf- 
level testing results are most likely to be substantial. In addition, fche study results indicate 
that instructional-level testing generally yields greater fall-to-spring grade-equivalent growth 
than the grade-level testing for the group of low achievers. Despite stfme methodological 
problems, these findings, if substantiated, would reinforce the concern for many evaluators 
of compensatory-education (CE) programs for the problem of selecting tests appropriate 
and sensitive to the examinees' achievement levels. ' v 

Summarizing the effects of out-of-level testing for the evaluation of CE programs, McLaugh- 
lin, Cilmartin, and Rossi (1977) concluded t>iat the'use of out-of-le\^l testing can appear to 
,have an effect independent of the effects of" the program being, evaluated. This can happen 
because &me l students simply store at the bottom of the norms, whether they take the 
grade-leyel or the below(-level test. Since the norms usually extend down further on the 
below-level test, the students' scores arentirmed lower than *they could be normed with 
the grade-level test. Thus, changing test levels can increase the apparent deficit of a student 
by a year or more (and it can decease that deficit in the same manner). The authors recom- 
mend that each student be tested with a test fevel that will provide a score in the mid-range. 
But heterogeneous^roups of students would require a test with several articulated levels 
and flexible administration procedures. — > t 



A FURTHER LOOK AT OUT-OF-LEVEL- SUHVEY TESTING 

v In order to implement the congressionally mandated evaluation of the sustaining effects of 
CE and of Title I programs in particular, we undertook a longitudinal study to describe the 

c nature and evaluate the outcomes of compensatory programs in a stratified-randorrv sample 
of 242 schools across the nation. At the time of the study's design, there was concern with 
previous studies of a similar nature that the achievement tests used were either at an inap- 
propriate level, thus providing insensitive if not invalid assessment, or that when achieve- 
ment was measured over test levels the inadequate inter-level articulation obscured the 
growth mdexe^ In order to overcome these potential problems, each of the more' than 
83,000 students inlfce representative sample in grades 1 through 6 were tested with two 
adjacent levels of the Comprehensive Tests of Basic Skills (CTBS) (Hoeptter and Christen, 
this volorae) during the first year of the study. * 5 ' 

Puring the fall of the first year of the study, each student was tested at the test level that the 
publisher recommended as the best leveMgra<3e-level), and was also tested with the level 
below' the recommended one (below-level). Thfs testing schedule would later enable the 
investigators to select the best functioning level of test for each grade at each school for 
later years of the study, and create a vertical scale of growth based uporvfewer assumptions 
than those of any published scales. In addition,, the data so gathered provide the base for 
the comparison of the effects on scores of grade-level and below-level testing, 

The Sample of Schools. From the 242 randomly selected schools, 111 schools were, 
selected so that most had all six grades in the 1 through 6 range and so that each cell of the 
sampling matrix wasYepresented by at least one school. The sampling matrix had 84 occu- 
pied cells with three stratification dimensions: geography, population density, and'poverty. 
As a check on the possible distortion from national representativeness in the final sample 
selection, the average percentage of poor students and tbe average percentage of minority' 
students per school were compared to projected estimates from a*large stratified sample of 
^ 4,750 schools (Hoepfner, Zagorski, and Welliscfvil977). In the large sample the projected 
* g#imates were 28.39 percent poor students per school and 23.79 percent minority students 
per school. In the reduced sample of 111 schools the average percentages were 27.23 and 
19.47, indicating a slight undersampling of schools with higher concentrations of poor and 
minority students. 

* e 

m 

For each school, two indexes were computed for each grade and each Jevel of test: .the* 
percentage of studen^scoring at or below the chance level (defined as the number of items 
divided by the number of alternative responses to each item), and the number of students 
With perfect or one-less-than-'perfe^t scores. These two indexes can clearly be seen to be 
• operationalizations of 'floor' and 'ceilingVeffects for the tests. 

The Questions Investigated. With the data for each grade and test leveLat each of the 
111 schools) several questions of interest were investigated. The relationship between the* 
percentage scoring below chance for each level would provide ar> answer to the question 
of whether below-level testing reduces floor effects. A similar analysis employihg the 
percentage of perfect and near-perfect scores answers the question, of whether below-leveL 
^ testing increases ceiling effects. Both answers, further, could be conditioned on the percent- 
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ages of minority and poverty students to provide better insights into the effects. The effects 
could also be investigated separately for reading and math scores, as both were available. v 

Because the test data had been completely re-qormed on a closely /Representative sample 
(the 242 schools) and vc^th inter-level scaling problems directly solved through the double 
testing procedure, the study was also able to provide evidence relative to t(|e assertion by 
Long et ah (1977) that the percentage of low-achieving students coidd be /manipulated 
systematically through the selection of grade-level or below-level testing. " 

'study findings ^ V • * 

Floor and Ceiling Effects by Test Level. The mean school percentages of students scoring 
3t or below the chance score, ,and of students with perfect or near perfect scores, are pre- 
sented m Table 13-1, by grade and by test level. It Is clear that in alfcases for both reading 
and math tests the use of below-level tests reduces the percentages of students scoring at or 
below the chance scorfc. the reduction is remarkably large at grade 1. Except for grade 5, 
the* grade-level tests resuit in an average of over 10 percent of'fche students scoring in the 
floor range. The ceiling effects are much less severe. In particular, .at grades 4, 5, and -6 
average percentages of students scoring at or near perfect are generally smclll (less than 2.5 
percent) with only one exception. At grades 1, 2, and 3 the ceiling effects are considerably 

Table 13-1 - * 



Average School Percentages of Reading and Math Scores Below the Chance-Level 

Floor and, at or Near the Ceiling, by Grade aVid Test Level « 



Grade 


Level 


Reading Scores 


Math Scores 


Average School 
Percentage 
Scoring at or 
Below Chance 


Average School 
Percentage 
Scoring at or 
Near Perfect 


Average School 
Percentage 
Scoring at or * 
Below Ohance 


Average School 
Percentage 
Scoring at or 
Near Perfect 


\ 

, 1 - 


Grade 


53.2 


1.0 


50.4 ' 


1 0.1 ' 




' Below 


10.5 • 


14.1 


2.6 


4.9 




Grade 


12.2 


3.4 


12.3 ' / 


^0.5* 




Below 


70 


15.9 


7.1 


5.2* 


3 


Grade - 


16.2 


0 1 




0.1 




Below 


2.3 • • 


16.4 

* \ 


*2.4 


6.3 


4 


Grade 

i.* 


14.1 


* 

0.0 


10.4 


(j.t) 




Be, low 


70 


0.6 




' • 2.3 


5 


Grade 


77 - / 


0 1 


' *± 


0.0 




Below 


2.6. 


2.2 


0.8' 


9.3 




Grade* 


* * 










'12.0 


0.0 




0 0. 


6 *K 


Below* 

tit 


3 9 


0.4 


1.7 


0.0 > 4 



smaller for math thaafor reading with belov\r-level tests. There are only negligible xeiling 
effects for all grade-level tests. tf ^ 

/As McLaughlin et al. remark, the floor effect is the more important afoblem, particularly 
/when testing CE students. In view of the present findings, r^school Specific recommenda- 
tions are not available, below-level tests of the CTBS are more likely to be appropriate for ' 
reading and math. , . % - 

Floor and Ceiling Effects by Poverty and Minority Concentrations. The schools* floor and 
ceiling percentages for each grade and for eaf h level of the reading and math tests were 
related to concentrations of poor and minority students. Although all the variables are ex- 
pressed^as percentages, their frequency distributions were generally quite skewed. Pearson 
product-moment correlations were nonetheless employed to show the 11 relationships 
because attempts to categorize the- variables into more arfalyzable ordinal 'form all too often 
pbscured the extent of the underlying relationship, the relevant average correlation'coeffi- 
cients are presented in Table 13-2. ■ 
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Table 13-2 



Average Correlations Over Grades of Extent of Floor and Ceiling 
Effects With ^hool Concentrations of Poverty and Minority Students 



% 


Poverty Concentration 


C v 1 
Minority Concentration 




Reading 


Math 


Reading 


Matb 


Extent of floor effects 
with grade-level tests 


65 


56 


.63 


— 

.51 


Extent of floor effects 
with below-leve^tests 


67- 


.49 * 


.62 X 


.44 

* 


Extent of ceiling effects 
witjh grade level tests* 


-.22 


- 16 


- ~ -.15 ' 1 > 


-.08 


Extent of ceiling erfects 
with below-level'tests; 


' -42 


- 39 


-\34 ' 


-.30 * 



'The average currelations for reading are pbtained tor grades 1 to 5 only, as no schools had any^rade 6 students 
coring at or near pertect tor 4he grade-fevel test In fact, there are non-negligible variations across schools in 
the extent of ceiling ettects\v*ith grade-level te^ts only from grades 1 and 2. For grades 3, 4, and 5, the variation 
is extremely small, with most schools having no students scyring at or near perfect Due to such a lack of varia- 
tiert'tind the extreme Newness ot the distributions, the low correlations are expected and de^not necessarily* 
\, reflect a lack ot, relationship between the extent of ceiling, effects and the concentration of poverty and minor- 
ity students. v " 1 ' 

• ■ . * < 

Likewise, the average correlations tor mrfth are obtained trom grades \, 2, 3, and 6 onlyras the ceiling effect in- 
dicator is ^ero lor grades 4 and 5 in all the schools. For all other grades, the variation^ are very smalL Again the 
distributions are highly skewed tor these grades. As a reslHt, no inference about the. relation between the extent 
ot ceiling effect with gracte level test and the.concentrations of poverty and mrnonty students shou!3 be made. 
Nevertheless, the average correlations are presented here Jor the completeness of the table. 




RIC 



The percentage of students scoring at or below the chance score correlates positively and 
substantially with school concentrations of poor and minority students, With both erade- 
level and below-level tests, these correlations are greater for reading than for math. A 
negative and smaller relationship is observed for the percentage of students scoring at or 
near perfect with below-level tests and' the school's concentration of poor and, minority 
students This relationship is'again slightly stronger in reading than in math. Since there are 
very small ceiling effects "With grade-level tests, the relationships -in these cases are of less 
interest (see footnote to Table 13-2): 

* 

In Urge-scale purvey testing, an etfaluator cannot ignore the floor effects— resulting from 
giving inappropriate test levels-on the sensitivity and validity of the data analyses. The 
results reported here suggest that knowledge of a school's concentration of poverty and 
minority can be utilized to aid in the selection of test levels to be administered. 

Between-Level and Between-Skill Area Correlations. Because there is concern with the 
problem of floor effects, it is of further interest to investigate the relationships between the 
floor effects of .reading and math tests, and between the floor effects of grade-level and 
below-level tests The average correlations between the floor effects of reading and math , ^ 
tests are .74 and 65 for grade-level and below-level tests, respectively. The correlations are ' 
comparable to the correlations between reading and math scores on the CTBS. The floor 
effects of the twoJ^vels of tests are also highly correlated, with averages of .74 for reading 
and .67 for math. The between-level correlations of floor effects are slightly lower than 
those of the test scores. 

Classifications of Student Achie\ement Status by Test Level. Jn order to verify the find- 
ings of Longet al. (1977), the nationally projected percentages of students who wQuld be 
classified as low achievers on the basis of scores one year or more below the expected level 
in terms of grade equivalents were determined ^parately for grade-level and below-level 
tests and also separately for reading and math. The percentages are presented in Table 13-1 

For reading, at every grade but the first, more students would be classified as low achievers 
by the grade-level tests than would be by the below-level test. For math, the same findings 
hold except that at grade 2, the below-level test appears to classify rrtore students as low 
^achievers. The difference, however, is so small that it is likely due to the discrke nature of 
the test scores and the rounding to integers. At grade 1, the cutoff for low-achieVing scores 
is approximately half a t year^ instead of the usu^J one year belownhe assigned grades (see 
footnote to Table 13-3 for explanations). The exception at the first grade could be explained 
by the severe floor effects of the grade-level tests and the necessity of extrapolating the 
grade-equivalent scale into the truncated-part of the score distribution. This phenomenon 
greatly reduces the percentage of studenfs classified as low achievers with the grade-level 
test as compared to that with the below-level test. 

The grade-equivalent scale was developed from, the vertical scale scores which, in turn, 
were calibrated by joint utilization of the test results from all grades and 1 both test levels at 
each grade Based on the grade-equivalent scale developed in this manner, the scores cor-* 
responding to the cutoff for the low-achieving category closely approximate the empirically 
determined performance levels of the typical students (i.e., median performer) from one 
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Table 13-3 



0 

Projected National Percentages of Studeitfs Scoring One Year or More 
Below Expected Grade-Equivalent Score, by Grade Level, Test Level, 
and by Reading and Math Scores 



Grade 




Percentage One Year or More Below Grade Level 




Reading Total ScAre 


Math Total Score 


Grade* Level 


^Below-Level 


\ Grade-Level 


Below- Level ^ 


y - ■ 


7 


14 


9 


13 




8 


6 


7 


8 


3 


2S 


17 


16 


15 


4 


31 


28 


20 


17 


5 


32 


31 


25 


18 


6 

• 


38 


33 


* 33 


27 



•For grade 1, the percentages are of students scoring approximately one-half year or more below the 1.1 grade 
equivalent Half a year is adopted because it corresponds to a similar percentile below which grade 2 students 
will have grade-equivalent scores one or more years below 2.1 in terms of the^rade-level norms. 

grade lower. Thus, the finding of more low achievers in terms of grade-level tests at grades 3 
to 6 cannot be simply attributed to the scaling procedure. Nor caQiJt- be accounted for by 
the greater difficulties of the grade-level tests. Rather, it probably reflects the skewed 
distributions of the test scores and the differences in tjie score distribution forms of the two 
test levels. « 

, The present results, based oi\the complete range of students, clarify the inconsistencies of 
the findings at grades 2, 3, and 4 reported by Long et-al. (1977). Thebresent data point to 
the conclusion that grade-level testing results in greater percentages of students considered 
as low achieving. Since trie grade-level tests would be expected to better reflect the content 
and the curriculum for most students tested, it would appear that grade-level tests can more 
accurately detect a student's low-achieving status, though they identify more low achievers 
in the population. 

v 

The data also confirm the claim by McLaughlm et al. (1977) that greater achievement deficit 
in terms of grade-equivalent units can be obtained with lower tost levels. This is largely 
because the grade-equivalent score corresponding to the chance-level score increases with 
test levels. A pronounced case can be^een at levels. 2 and 3 of the CTBS. It was found, 
based on the* new norms, that if a sixth grader obtains the chance-level sccfres oh both test 
levels in reading, a grade equivalent of 3.4 will be earned with level 3, but only 2rA willl^e 
earned on level 2. Consequently, the student will be judged to be one mo/e/ear behind 
the level 2 test. In this sense, grade-equivalent units in general are not ideen measures of 
achievement deficiency. 
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263 

238 



CONCLUSIONS 

• 

This study reveals that the use of below-level tests reduce* the percentages of students scor- 
ing below chance levels. It also reduces the percentage of students scoring one year or 
. more below their expected grade levels, with the only noticeable exception at grade 1 The 
floor effects-were found to be greater in schools with higher concentrations of poor and 
minority students. The ceiling effects, on the other hand, were small for all grade-level tests 
With below-level tests, considerable amounts of ceiling effects were found at grades 1 to 3 
The relationship of below-level ceiling effects and the poverty and minority concentrations' 
is negative and weaker than "that observed for the floor effects. r 



TAPTER 14. SPEEDEDNESS OF ACHIEVEMENT TESTS IN THE 
SUSTAINING EFFECTS STUDY 



Ralph'Hoepfner 

, The test of academic achievement used in the Sustaining Effects Study (SES) was 
* examined for differential effects of speededness among racial ethnic groups. 
Although small differences in speededness were detected, it is not at alt clear that 
the differences result in bias in the measurement of achievement that will later 
have a detrimental impact on study findings. The' reason for this, it is argued, is 
• that there is no compelling argument for the exclusion of speed of thinking and 
responding in measures of the outcomes of compensatory education. 

Because the was expected to pro.vide answers to questions of educational policy, it was 
planned from the beginning that shortcomings of previous evaluations would be avoided. 
For this reason, a great deal of care went into the selection of the tests of academic achieve- 
ment (see Hoepfner and 'Christen, this volume). Sufficient time and effort were allocated to 
ensure that the achievement measure wouid result, as much as possible, in clear finding?. 
The study; for example, built its own fall,and spring norms, so that indexes of growth would 
not depend On interpolations or extrapolations. Vertical-scale scores were developed for 
the-study that Capitalized on the semi-annual admigistra^ions and that did not depend on 
assumptions oT equivalence between different groups of students (Hemenway et al„ 1978). 
Items exhibiting statistical bias were removed from the scores (Hoepfner and Christen, this 
volume). In' other words, many of the traditional assumptions or practices of achievement 
testing were notf uncritically accepted. 

In part, this critical approach was based ch concerns regarding how the achievement of 
deprived students can be fairly assessed, because the focus of the study was to be the 
achievemeot growth of (what turned out to be) TUIe I students over more than one school 
7ear. Problems of item bias and out-of-level testing were faced and resolved early (Hoepf- 
ner and Christen; Hoepfner and W^ng, this volume), as there was a history of concern with 
them. The issue of the effects of. speeded tests on the results for deprived students had riot 
enjoyed such historical concern, but the issue was raised early in the study, and it had to be 
resolved prior tq jhe' planning of test administrations. * . * 

CONJECTURES ABOUT SPEEDED TESTS 

The lack of previous empirical findings in this matter led severaTadvisors to the study to 
conjecture regarding how the use o{ speeded tests might bias or obviate the findings. These 
conjectures can best be understood as arising from issues of bias, validity, and logistics, 

V* J 

Speeded Tests and Biased Measurement. Standardized achievement tests have been 
developed so that the established time limit permits the typical student' to attempt or com- 
plete all the items. Most definitions of test speededness assume that some small percentage 



of examinees will not have time to complete or attempt all items. If this small percentage \s 
non-randomly distributed .among certain groups of students, then conditions of speeded- 
ness introduce bias against members of those groups. 

It was suggested that disadvantaged or deprived students may' be just such a group. If speed- 
ed* tests showed greater socioeconomic or even racial/ethnic differences than unspeeded 
tests do r such information wc)uld lend credence to .the suggestion. In this'event, the study 
would have to take strong steps to reduce or eliminate the biasing factor so that the conce- 
ptions* drawn from the study, concerning the effectiveness of programs designed for those 
deprived students, would not be biased in a manner that reduces the observed growth and 
leads to a conclusion that the programs are not effective. 

The logic behind the supposition that scores fjom timed tests can contribute to bias is that 
the rate and accuracy of responding depend on,cultural, personality, and motivational 
factors as well as on ability. Inaccuracy were equally distributed among racial/ethnic' 
groups, but rate were not, then a timed test would* be biased against thQse grogps with 
slower rates, even if they had equaf accuracy. The slower rafes would result in. lower total 
scores on timed tests, thus penalizing the slow but acoffatei responder. Khan°(1968), study- 
ing high school students, found that time limits influence scores of black students to a 
greater degree than white students. He concluded that speededness may bring out cultural 
diffpences, and may, therefore,, reduce the validity of a test. Because Title I students are 
disproportionally minority students (Wang, Hoepfner, Zagorski, Hemenway, Brown, and 
Bear, 1978), we might expect'speeded tests to distort their actual achievement. 

t 

On the other hand, Berry and Lopez (1977), in a comprehensive review of testing problems^ 
for Spanish-speaKing students, failed to mention speededness as a culturally biasing factor. 
When Flaugher and Pike (1970) administered* tests that were too difficult to inner-city high 
school students, although scores were, very low, measures of speededness were similar to 
those based on a nationally representative sample., Their findings strongly suggest that diffi- 
cult achievement tests are not differentially speeded for minority groups. Attempting to 
reduce expected bias caused by speededness, Evans and Reilly (1972, 1973) found that 
•scores for both black end white college seniors increased, but not differentially. They con- 
cluded that reducing speededness irUests was not. beneficial to the black exammeesjnsofar 
as the increased time did -not reduce score differences between tfie minority and majority 
groups Using only bright students, Bridgeman (1980) showed that students wh<\are quick 
at one task may not be quick at others, that there is no single and general trait of quick- 
ness that can, at present, be ascribed to any "group of individuals, rit) mattep hc5w they 
<jre characterized. * . v 

Yates (1966) found that some nine-year-old children's scores are more seriously affected by 
speededness than others, but no personality factors could be found associated with those 
differences. Likewise, Wasson (1969) found that extending* the time limits for a reading test 
for fifth-grade students resulted in increasing total scores, but the increases were not 
associated with characteristicsJSuch as tested intelligence, o/ even reading speed. Miller 
and Weiss (l976) t found that examinees of different ability levels show~ similar patterns of 
response in adapting to timed or untimed testing conditions. ' *• 
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The findings and conclusions from these earlier studies, while not confirming or denying 
biasing effects "of speeded test conditions, suggest that the SES ought to investigate them 
further, so that the issue might be resolved and tfie study thereby improved. 

, Speededness and Validity. Use of speed-dependent outcome measures could affect both 
the internal and external validity of a study. To the extent that the degree of speededness 
varies among test forms and administrations, estimates ?f actual growth will be influenced. 
The results are confounded even more if the test scores from the -groups fc^ing compared 
are characterized by "differential speededness. Either of these occurrences could affect in- 
terpretations of the' results oRhe study. 

"With respect to the effects of speededness on the external validity of the study, we must 
question the extent to which speeded performance is the goal of compensatory education 
programs or, more generally, if such performance will help the students in such programs 
to rise out of their deprived conditions. As long as the instruction is speeded, the learning of 
students will depend on mental quickness. (This offers a partial explanation of why tests are 
su£h valid predictors of academic performance.) However if speededness is removed fcom 
the instruction, then a speeded test will be less valid as an indicatoc of effectiveness^ 
instruction To the extent, then, that quickness of response is not something that willjm- 
prove as a result of CE programs, it should not be part of the indicator of its effectiveness. 
(One could question, further, if mental quickness ought to be at least'an implicit effect on 
the grounds that it may be important for the students "in functioning in the non-academic 
job market they are likely to try to enter, but such questions seem highly speculative.) 

Psychologists have not definitively resolved the problem of how speededness should- be 
considered in relation to intellectual performance. We simply have jio supportable and 
general arlswers to questions regarding the possibility of thinking slowly but well, if slow 
performance is mainly a reflection of long latency in initiating thought on a problem or 
slowness in plowing through the stages of a solution, if it is caused by checking and re- 
checkmg possible solutions, orlf it merely indicates slowness of responding after a solution 
hasten achieved. Horn (1979) found that the correlation between number-correct scores 
and speedmess of response is generally low (about +.22) and he concludes that they are 
largely trvJependent. His data imply that slowness in providing answers is not necessarily 
indicative of providing poor answers. Speediness, of course, is a factor that appears to run 
^ through all forms of intellectual and performance decline witJi ^agmg (Birren, 1974), 

Speededness and Testing Logistics. The conditions under which a test becomes speeded 
primarily include administration under rigid time limits. Variations in the time limits will 
, result in differences in mean scores. Since it is very difficult to achieve uniform adher- 
ence to time limits, one could consider untimed tests in which each student is alfowed as 
much time as needed to work on the test (see Rindler, 1979, for other approaches to as- 
sessing speededness). 

This univer^solution to the problem of speededness would eliminate the suspicion that 
v some test-wise teachers manipulate testing time to influence their students' scores. Whether 
or not teachers of deprived children allow them a bit extra time, or whether they reduce 
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time for pretests and increase itlfor posttSstsjoj whether they reduce it td improve chances 
to qualify for special compensatory services, is a matter of speculation. Granting each stu- 
dent the time needed would ehmirtate jthose confoundmgs, but would present logistic 

• problems^ test administration At the ages where test items are read aloudigenerally grade 
2 and lower), the pupils could raise 'th'e.ir h^nds when they have responded, so that the 
students, not the administrator, set ths pace But just how patient would administrators- be 

. before they made comments designed to hurry the slow students along? One also has to" 
consider jhe peer pressure of the group that is waiting foe the next item, t* say nothing 
tired arms and wandering attentions.. v . * 

At the higher grades, it may il? difficult to keep the faster students ^occupied and well- 
behaved under truly unspeeded conditions. Assigning them additional test items may be 
seen as an unfair burden on their cooperatfon. It shoutd be clear, there/ore, .that the solu- 
tions to the potential problems of speededness may oot provide a net gain for the SES. 

THE SPEEDEDNESS OF THE ACHIEVEMENT TEST FOR THE SES 

The technical reports iCTB/McGraw-Hill, 1974) that support the Comprehensive Tests of 
' * Basic jSkills (CTBS;. the test chosen as the measure o$ student achievement for the study, are 
not specific about the degree of speededness of the tests or if thfe speededness is different 
for any identifiable group of students The publisher did supply us with data tapes selected 
from its normmg sample that provided us the item response s<%ed as right, wrong, or omit- 
ted tor each test level and grade. The item data for each student w,as also associated with a 
racial'ethnic identification of the 1 student into ,tyack, brown, and other ipredommantly 
white) By tabulating the mefdence ot omitted items at th^end of* the test/ we could deter- 
mine if any of itf test levels were speeded. The approach will give* somewhat inflated 
estimates of spe^ednes^, however, because students are not encourage^ to guess on 
♦terns, and some of the emitted responses may be due to lack of knowledge and reluctance* 

* to guess, instead of insufficient time to address the item. '* 4 " 

Speededness, of course, is a relative thing. In order to give it some specific meaning criteria 
ha^£ been set by which one qan classify, a test as speeded., These criteria generally use one 
or more of the following test Characteristics. (1)«the* percentage of examinees attempting the 
last item of the test, (2), the percentage^ examinees completing the first 75 percent of the 
test items, and (3) the percentage of .items attempted by at least 80 percent of the exam- 
^ inees. The tabulatiQns based on a random *ample*of the publishers data tape.are presented 
in Table 14-1. The reader should keep in mind, however*, as noted above, lhat the values in 
* t . Table 14-1 probably over-esjjmate the speededness^of the tests. ► 

" The values in Table 14-1 indicate that Speededness is greatest for the Math Computation 
scales. Tests of th*s kind are frequently speeded, in part because the domain of*items is so 
large and the items so easv to'create, and in part, presumably: because speed of calculation 
does have some practical v^lue The Ma1h Concepts scale i^ least speeded. Speededness 
appears greater at grades 4 through 6 than at the earlier grades, but, this change does not 
parallel the change from oral to silent administration of the* scales. Although by most stan- 
dards the data in Table 14-1 would be interpreted as indicating that the CTBS scales are not 
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Table 14-1 



Three Indexes of Speededness of the CTBS Scales, by Grade 



Scale* 
Grade 


Percentages 


Percentages 


Number of Items. 


Attempting 


Completing the^First 


Attempted by at least 


the Last Item 


75% of Items** 


80% of Examinees 


Reading Vocabulary 








C An 1 

Grade l 


96.2 


98.9 


10P.0 ' 


Grade 2 


96.6 


97.2 


100.0 


r* -An ^ 

e Grade 3 


* * 99.2 


99 9 


100.0 


Grade 4 


86.3 


90.2 * 


100.0 


Grade 5 


' 86.2 


92.5 


100.0 


Grade 6 


82.0 


90.2 


94.2 


Reading Comprehension 








Grade 1 


96 4 


9/ o 


17)0.0 


Grade 2 


v 86 1 


90 0 


100.0 


Grade 3 


94 5 


94.7 


100.0 


Grade 4 


88 1 


91 o 


99.2 


Grade d 


86.8 


92 6 


100.0 , 


Grade 6 


B6.9 


93.6 


100.0 


Math Concepts 








Grade 1 


90.7 


98.2 


100.0 


Grade 2 


96.7 


97. j 


100.0 


Grade 3 


94.5 


98.0 


100.0 


» Grade 4 


95 5 


96. o 


rtO t 

98.7 


Grade 5 


97.4 


. 98.4 


100.0 


Grade 6 


96,9 


98.3 


100.0 


Math Computation 




i 




Grade 2 


* 82.5 ' 


v 87.5 


100D 


Grade 3 


95.6 


97.8 


100.0 


Grade 4 


92.7 


95 7 


-100.0 


Grade 


78.2 


87.4 


m 

%.5 


Grade 6 


81. 5 ' 


91.4 


' 95:8 


. Weighted Totals 








Reading Vocabulary 


87.9 v 


93.1 


97.6 


Reading Comprehension 


88.4 


93.2 


99.5 


Math Concepts 


96.0 


97.9 


99.fi- 


Math Computation 


842 


91.4 


97.6- 



€ • 

•The Reading Vocabulary scales in grades 1 and 2 are named Sound Matching' and 'Word Recognition 11/ 
respectively, the Reading Comprehension scales in grades 1 and 3 are Letter Sounds' and Comprehension. 
Passages.* At grade 1, only one math scale is given, Xamed Mathematics/ which we have classified as a Math 
Concepts scale for this table and for Figure 14-3. 



**Tbe percentages reported »n this column are (he smallest percentages of omitted responses for^y item in the 
last fourth of the scale's items. ' T 
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speeded, there is some speed which may have more than a negligible impact on study find- 
ings if it is distributed in a manner to confound analysis. 

DIFFERENTIAL SPEEDEDNESS AMONG RACIAL>ETHNIC GROUPS 

The students from- the publisher's norming group were not identified by their participation 
in compensatory programs, so direct tests of speededness could not be made among the 
comparison groups to be used in the study. JWiere was information on the race/ethnicity of 
the students on which differences in speededness could be observed, however, so the" 
percentage of responses omitted by students in each racial/ethnic group was tabulated for 
each item of each scale. In order to make the rates of omitted responses visually compara- 
ble, the items in each scale were calibrated so that all scales would appear to have 100 
items. Then the .plots of the percentages of omitted responses by item were smoothed by 
the, method of moving averages. The results are presented in Figures.14-1 through 14-4, for 
each of the CTBS scales. • 

The important things to look for in the4igures are curves that rise at different rates for the 
three groups {if the curves are level or declining, the omissions are not likely due to condi- 
tions of speed, but to disinclinations to respond). In grades 1 through 3, we can see con- 
firmation of the data in Table 14-1: speededness is not apparent, but response omissions 
occur for all the items. At these grades, the 'other' students have lower rates of omitted 
responses than the black or brown students, but the differences in rates are frequently not 
large. Except for the Math Concepts scale, speededness becomes apparent in the higher 
grades. In general, we can conclude that speededness- is most pronounced for black 
students at the higher grades and less pronounced for brown students. . 



The differences in rates of omitted responses become quite large at limes (as-much as 15 
percent), so difference^ in the observed scores will be noticeably influenced. by them. 
Because it seems safe to assume thatjtbedifferences in rates of omissions are largely due to 
speededness, we Conclude that speededness will play a role in the analytic comparisons of 
the study, This role will be limited to the correspondence between the racial/ethnic grpups 
and the comparison groups of students formed on the basis of their participation in CF,' pro- 
grams, 'but it is difficult to assess exactly because while greater 'percentages of minority, 
students participate in CE programs, more majority students, in absolute numbers, partici- 
pate Our conclusions regarding the meaning of the effects of speededness still depend on 
whether we accept test-taking speed as a reflection of mental speed, and then whether we 
believe that n*ntal speed is a desirable outcome of compensatory services. To the extent 
-that we do brieve it we have increased confidence in the external validity of the measures 
of academic achievement. 



i 

4 



2/() 



ERIC 



246 



28% - 



•21% - 



14% 



7% - 



0% 



Grade 6 



Black students . 
Brown students 
Other students 






Grade 5 










18% 


> 










12% 


< 










6% 












0% 













Grade 4 






" " 0 




— 



16% 
12% 
8% 
4% 

0% 

4% 
2% 
U% 

8% 
4% 

' 0% 



Grade i 






1 " " I ^ B 1 1 * ' * • • • • ♦ • «tt^ • fr* it; . . . 










Grade 2 



















Grade 1 
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Figure 14-1 





Smoothed Percentages of Omitted Responses (Vertical Axes) for Items of the 
CTBS Reading Vocaltffery Scale (Horizontal Axes, Scaled so All Test Levels 
Appear To Have the^Be Numherof Items) for Three Racial/Ethnic Groups, 
flAs Indexes of Speededness. 
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16% 
12% 
8% 
4% 
0% 

8% 
4% 
0% 



Grade 4 




_L 



Grade 3 




Figure 14-2 • 

Smoothed Percentages of Omitted Responses (Vertical Axes) for Items of the 
CTBS Reading Comprehension Scale (Horizontal Axes, Scaled go Ail Test Levels 
Appear To Have the Same Number of Items) for Three Racial/Ethnic Groups/ 

as Indexes of Speeded ness. 

, • 272 
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5% 
4% 
3% 
2% 
1% 
0% 

5% 
4% 

A 

2% 
1% 
0% 

12% 
10% 
8% 
6%, 
4% 
2% 
0% 

8% 

6% 
4% 

2% 

. 0% 




Black students 
Brown student* — — - 
Other students 



Grade S 



Grade 4 




f 





I I . I 
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Grade 5 




8% 
b% 

4% t 



2% 

0% 



Grade I 




20 



40 



70 




Figure 14-3 



Smoothed Percentages of Omitte<J Responses (Vertical Axes) for Items of the 

CTBS Math Concepts Scale (Horizontal Axes, Scaled so All Test Levels} 
Appear To Have the Same Number of Items) for Three Racial/Ethnic Groups, 
• as Indexes of Speededness. 
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Grade 6 




12% 

4% 
0% 



Grade i ' * — / * 




..-j'S*' 4 *'" 1 ? 

*= — H r- 1 S ^ 1 ' - 1 


L f _ t1 r 



Grade 2 




* 



♦80 



90 



Figure 14-4 



v 



■0 



Smoothed Percentages of Omitted Responses (Vertical Axes) for Items of the 

CTBS Math Computation Scalfe (Horizontal Axes, Scaled" so All Test Levels 
AppearTo Have the Same Number of Items) for Three Racial/Ethnic-Groups, 
, as Indexes of^Speededness. 

C • - * - * : 
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PART IV. THE EJECTS OF ATTRITION 



Part IV is composed of two chapters that address two issues of attrition of cases from the 
longitudinal, study. In the first chapter, the reduction ig the study's sample of schools is 
described. Although the retained sample was selected with specific purposes in mind, statis- 
tical comparisons show its ability to represent the nation very well. Some potential compli- 
cations for the study analyses resulting frQm the systematic sample reduction are discussed. 

Based on the first full calendar year of the Sustaining Effects Study, we document, in the 
second chapter, the incidence of attrition and compare rates over several dimensions- of 
student chracteristics, such as minority status and achievement level. We also^tudy the 
attrition rates by several characteristics jointly, in order to understand better We major 
sources of attrition. Finally, 4ve provide some conjectures about the expected influences of 
the observed attrition on different kinds of analyses and issues in the study. 



CHAPTER 15. THE REDUCED SAMPLE F£R THE SECONDVND 
THIRD YEARS OF THE SUSTAINING EFFECTS-STUDY 



Ralph Hoepfner 

The representative sample of schools for the first yeai of the Sustaining Effects * 
Study (SES)^was created large enough, to ensure accurate national projections of 
the incidence pf Title I, sojjyat Congressional mandates for highly accurate hp* 
tional estimates could bemSt. In the second year of the study, when national 
projections were no longer planned, the sample was reduced. Sample reduction 
was accomplished in several ways th&t could have resulted in biases that might 
influence the analyses of the^ongitudinal effects of Title I. Therefore,, several tests 
were made of the reduced sample to assess the bia$. The results of the tests indi- 
cated that^evec though the sample reductioihrad not.beeniraftdom, the reduced 
sample was .representative of the nation's schools and could be expected to 
support inferential -analyses with very little distortion: ' $T ^ m / 

The complete /irst-year sample for the SES yvas composed pf 326 school^. It was composed 
of one nationally representative sample, randomly selected, and three purposive samples, 
selected to prQyfde specific kir*ds w of schools for comparative analyses. The four sijbsamptes 
(described in detail iri Hoepfner, Zagorski, and Wellisch, 1977) were as follows: 

• 242 randomly selected representative.schools, ' ' ^ 
14 school^ that fed into orTrom the representative schools, 

• 43 schofofs nominated as having promising compensatory programs, and 

* * N ° • 

• 29 schools «with high poverty but nQ compensatory funding. 

* ' * * •* * 

The size of the representative sample was necessitated by the need to make highly accurate 
population projections about the' state of the nation's compensatory education (CE) efforts. 
An even more critical need for'thfc representative* sample arose from the congressional 
mandate to.ascertain the numbers of children who are and are N not being served by Title I 
who^are poor and fow achieving. Arriving at accurate national projections 4o* provide 
Answers to Congress' questions required th^ large first-year' sample. 

After the first year of the study, descriptive projections to the nation wepe no longer planned. 
Instead, the study was to focus on inferential comparisons among vari6us types, j^proaches 
to,pnd intensities of compensatory education iri terms of their effectiveness. .With the 
inferential goals in mind, it was necessary to maintain- the three purposive samples in order 
to provide the variations needed for the analyses. But it was no longer necessary to main- 
tain the large number of representative schools. The schools to.be 'retained for the second 
year of the study were, of course, not to be atypical or nonrepresentative, the inferential 
requirements of the study simply assumed greater importance than the descriptive ones. 4 
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On this basis, 177 schools were retained for the second year (two schools left the study in 
the third year): _ ^ 

• 95 representative' schools, selected p^rparily so that each cell of the first-year sampling 
matrix was represented by at least one school, but selected secondarily within cells 
with a systematic preference for schooJs with summer programs, with high percent- 
ages of low-achieving or poverty students, and with all six elementary-level grades; ^ 

• 13 schools that feed into or out of the representati^tchools selected for the second 
' year; ' $ 

% 41 schools nominated as having promising compensatory programs; 

• 28 schools with high poverty but no a^rensatory funds. 
THE REPRESENTATIVENESS OF THE REDUCED SAMPLE* 

, After the sejection of the reduced second-year sample, concerns were expressed tbat it 
might have become so unrepresentative, due to the selection criteria, that it would not be 
capable of supplying statistics that represent the nation's CE efforts. The first step in ascer- 
taining whether this concern is justified is to inspect the distribution of the s§cond-year 
sample over the sampling matrix for the first year.' Because the samplin&jTiatrix provides 
cells* that can be weighted to national projections, and has been found to provide highly 
accurate projections (Hoepfner et al., 19777, it is important to verify that the second-year 
sample has reasonable numbers of schools representing each of the strata, and does not 
Vhave additional missing cells' (Exact prop6ftionality of cell frequencies is not critical, how- 
; ' % ever, because the comparative analyses merely need* general representativeness, and 
weighting procedures could be used to adjust for disproportionalities if projections' were to 
be made.) v " • ' 

* The criteria for selecting the second-year sample guaranteed that -no (fells of fhe sampling 
matrix would 'be empty. Table 15-1 provides the unweighted distribution of schools accord- 

. ing to. the levels of each of the three sampling strata. It can be seen from Table 15*1 that 
eac^of (he levels of e$ch of the strata is well represented. The I^Vels of the geographic 

' region *stcata*are still close to being proportionally represented, so that unweighted data 
'would not have troublesome regional biases, and so^that appropriately derived weights 
would provide a close approximation to" pie distribution of the population, The levels of the 
district size and district poverty strata are similarly represented. Table 15-1, however, 
presents information that merely assures u$>that,ths sample is not seriously distorted. arid 
wodd be capable of providing information that could be weighted, it does not indicate that 
the weighted information would necessarily be accurate estimates of the population'. 

* ♦ * 

The question remaining is whether the specifically selected sample 'can effectively support 

•the comparative inferential investigations of the study. Clearly, inferential statistics Such as' 

^ L L 0r ! are n ot <*s vulnerable to devjations from representative and random sampling as are 

population projections, But extremely biased sampling can. have a biasing effect on thfem. 

' •• , - 277: • > 
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' Table 15-1 . 

Percentages of Public Schools With Grades in the 1-6 Range, 
by Str^ind Level of the Sampling Matrix 



Sampling Dimension. 

: 0 — 


*- 

Pnni il stLrkn r\i 
rupuitfliun Ui 

the Nation's 
\ 62,534 Schools 
v With Grades 1-6 


Respondent 
Sample of 
4,750 
Schools 


First-Year 
Representa- 
tive ^amnlf 

of 243 
Schools 


Second-Year 
Representa- 

live 3aiiipn. 

of 95 

Schools 
—* — 1 ' 










* 


New England 


7.10 


J.QO 


Q ft A 


y.4/ 


' Metropolitan Northeast 


8.32 




9.47 


• 9 47 


Mid-Atlantic 


10.59 


11.87 


K).70 


. 10.53 


Southeast 


44.58 


20.72 


10.29 


10.53 


North Midwest • 


17.53 


16.23 


11.93 


10.53 


South Central 


15.47 


13.56 


11.>1 


9.47 


Central Midwest 


7.11 


5 98 


J1.11 


10.53 


NJrtrtk C antral 

i^onn v^enirai 


5.20 


5.03 


8.64 


9.47 


raciiic ^ouinwest 


9.87 


10 00 


9.88 


12.63 


Pacific Northwest 


I 4.23 


3.35 


7.00 


7.37 


District Size 










Small District- 


45.51 


33.24 


36.21 


35.79 


Medium District 


30.41 




32.92 


29.47 


J Large District 


24.08 


33.16 


30.86 


34.74 


* District Poverty Level 










.High Poverty - 


22.05 


33.41 „ v 


29.22 


28.42 


Medium Poverty - 


29,% 


32.63 


34.57 


34.74 ^ 


Low Poverty 

1 : ■ ^ 


47.-99 


33.% 


36.21* * 


' 36.84 



The effects of*any nonrepresentativeness of the sample on the analytic outcomes, according* 
to Kish (1965 r pp 595-597) are not yet precisely identifiable, but there is a consensupTha^ 
the inferential statistics planned for the comparative analyses are not seriously distorted by 
small deviations from true representativeness of the. sample. Because we nonetheless did 
not want to push our, luck in this hotter, the second-year sample was selected in a manner 
that was expected to maintainj^f^resentativeness while maximizing our control of impor- 
tant independent variables fe g , poverty l£vel of school, availability of CE program by type 
and funding source).^ / 

THE ADEQUACY OF POPULATION PROJECTIONS BASED ON THE REDUCED SAMPLE . 

•Ugically, jt is difficult to Show that a sample is not unrepresentative in al( important aspects. 

^V/e elected to play the devil's Advocate and test some projected statistics, even though we 
had no plans td use projected data for the comparative analyses. Our thinking went some- 
jhing like>his sampling weights wjll oot make a bad sample look good-thefy are inferential' 
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aids, not cosmetics. Therefore, if we weight selected data (not the sampling dimensions of 
the previous section, .which if weighted merely show whether we can calculate with accu- 
racy) from the reduced sample and compare those projections to data in which we have 
confidence, we indirectly test how good the sample is. In other v^ords, if the projections 
are close to what we expect, we are safe in inferring that the reduced sample is not seri- 
ously distorted. ' > 

Further to determine precisely hovy accurate population projections based on the'Wond- 
year sample actually are, it isN?ecessary to make those projections based on newly calcu- 
lated weights, but on data collected in the firsryeaf. In this way, the projections can be' 
rigorously compared to known estimates with no contamination by history. Nine critical 
variables were selected as characteristics to be- projected to test for accuracy. The nine 

characteristics are: • * 

* t - 

1. Urbanism of school 

2. School and grade enrollments. ' 

3 Student poverty * ' * 

4. Student minority concentration of school • * 

5. Student race/e*hnicity 

6. b School concentration of low-achieving. students A 

s 7. Three-year stability (non-mobility) . 4 

8. Grade span type of school • > 

9. School CE funding 1 

The nine characteristics are projected from the 95 schools in the second-year representative 
sample and the projections are then compared ^o two other projected values*. The critical 
comparison is with the projections from the Principal Sample of 4,750 schools surveyed 
during the planning year_ofjtJ^tudy. This sample was fand^mly selected and was suffi- 
ciently large so that error estimates Were extremely small. The sample was tjsed as a test of 
the accuracy of the projections made from the 243. representative schools (one of the 
schools later left the study, leaving 242 for the entire first year) in the first-year sample 
(Hoepfner et al., 1977)': The projections- from the first-year sample aj£-also shown in the 
tables that follow for purposes of completeness. In addition, census information. on raceL 
ethnicity is also presented as another comparison to te& for the accuracy xif tjje second^ 
year sample projections! * 1 *\* 

Urbaptsm. of School. Schools were classified into four categories of population density 
and percentages o( schools falling into each category were appropriately projected fpr eacb- 
of the three samples. The result of the projections are presented.m Table 15-2. The second- 
feaj sample's greatest projection error is 5.3 percent, an underestimate for small cities and-, 
rural areas near cities An overestimate bT 5.8 (*rcent for medium-sized cities is in an adja- 
cent category Projections of percentages in ferge cities and Jn rural areas not near afces Sre ' 
accurate to less than .8 of one' percent. If national projections were made torn the (educed 
sarpple, we would expect them to be biased toward areas qf higher population density. The 
distention \rfpuld, of course, be caused directly by the selection preferences for schools (see ' 



Table VII-7 for the relationship between urbanPsm and poverty; Table VII-19 for the rela- 
tionship with poverty; and Table VII-45 for the relationship with incidence of summer 
school; Hoepfrrer et al., 1^77). • ' \ s # 

School Enrollment Enrollment projections were made by enrollment categories and by 
absolute student counts. Table IS^presents the enrollment distribution by five categories 
of school size*and Table 15-4 presents projected mean values of total school enrolment 
and of enrollment by grade. 

In all five enrollment categories the seconder s^nple projections are within 1 percent pf 
Mhe projections of the Principal Sample. Th^Hcond-year sampled fact, appears to be bet- 
ter in this respect than the first-year sample from which it came. 



Table 15-2 

Projected Percentages of U.S. Public Schools With Grades 
in the 1-6 Range, by Urbanism of School 



Principal First-Year Second-Year 
Uranism ^ Sample of Sample of. Sample 
^ • \ 4,750 243 ?5 4 
- \Schools Schools - Schools * 

' % 

City of over 200/000 1 f.2 10.8 1 2.0 

City of 50,000 to 200,000 or suburb - 25.4 * 25.7 29.2 

Coy under 50.000 or rural near city 38.5 36.4. ' 33.2 

Rural area not near city 24.9 * * 27.1 * 25 6 



Table 15-3 



^Projected Percentages of U.S. Public Schools With Grades 
in^the 1-6 Range, by Enrollment of School^ 



Enrolment In 
Grades 1-6 



Principal 
Sample of 
4,750 
Schools 



First-Year 
Sample of 
243 
Schools 



Second-Year 
Sample of 

95 
Schools 



1-200 
201-400 
401- 600 
661-800 
801-2,800 



28.4 
40.4 
21.4 
70 
2 7 



34.1 
40.1 
18.5 

2.0 



29.4 
40.6 
•21.5 
6.7 
1.8 



FrorrTthe projected enrollment numbers in Table 15-4, it can be seen that the second-year 
sample is composed of slightly larger schools than in either of the comparison samples. The 
'larger enrollments .are primarily at the primary grades, while the upper grades in the 
second-year Sample are somewhat smaller (see Table VII-45 in Hoepfner et al., 1977, for 
the relation between ^enrollment and the availability of summer school). 

Student Poverty. Three, categories of poverty concentration of the schools were em- 
ployed to test the projections of ^e second-year sample. From Table 'l 5-5 it can be seen 
that the projections of the second-year safnple over-estimate the high-poverty category by 7 
percent and.underestimate the lower poverty categories. These*errors are directly expect- 
able outcomes from the secondary selection criterion of poverty that was used for the 
second-year sample. 



Table 15-4 

Projected Percentages of U.S. Pyblic Schools With 
Grades in the 1-6 Range* 



Enrollment 
Averages 




Principal 
Sample of 
4,750 * 
Schools 


First-Year 
, Sample of 
243 
Schools 


Second-Year 
Sample of 

95 

Schools / 


Sch66i Total 




333 93 


320 05 


342 40 


Grade 1 




62 82 ' 


59 79 


63 87 


* jGrao^ 




58 74 


55 21 * 


60 19 


C/ade 3 


58 91 


: 54 53 4 


60 33 


/Crade 4 






, 55 91 


'59 71 


" CVade 5 




P> 65 95 


6) 47 


64 60 


^rade 6 


V 


"5 63 
1 


73 01 . . 


" 6*8 29 



Table 15-5 



Projected Percentages of U:S. Public Schools With Grades 
"» in the 1-6 Range, by Student Pqverty Level 



» 

Percentage of Students 
Below Poverty 


Principal 
Sample of 
. ' 4,750 




First-Year 
Sample of 
243 


SecjDncfeYeir 
' Sample of 
' . 95 




*^ 


Schools 
I ^ 




Schools 


Scjiools 


• 


, 0-20 


51 6 




.50 0 


* 46.5 




21 50 . ,» f 


32 3 




31 7 


30 4 




„ ' 51 TOO . 


16 I 




' 18 3 


23 1 





c 
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Minority Concentration. , Projection* from the second-year sample of percentages pf 
schools with each of four categories of minority concentrations are accurate tp less than 
2 percent The second-year sample projections tend to underestimate the low-concentra- 
tion categories and to overestimate the high-concentration categories, cleady the result 
of the secondary election criteria ot poverty and low achievement used for'the second- 
year sample. ' < 

Race/Ethnicity . On the^ basis £>Mhe projections from Table 15-6, it would be expected 
.that some racial/ethnic groups wiH be overrepresented in projections from the second-year 
sample. In Table 15-7 it can be seen that the greatest overrepreseritafion is of the Black 



Table 15^6 



Projected Percentages of U.S. Public Schools With Grades 
in the J -6 Range, by Minority Concentration 





Principal 


First-Year ' 


Second-Year 


Percentage of 


Sample of 


Sample of 


Sample of . 


Minority Students 


4,750 


243 


95 


* 


Schools 


Schools 


Schools 


0-19 


68.4 


o 

71 0 


66 1 


2049 


16 4 


156 


153 


SO- ^9 


- "> 


6 6 


9 1 


80-100 


' 80 


68 


9 5 



✓ . - Table 15-7 

Projected Percentages' of Students Enrolled in U.S. Public Schools 
r-s. With GradHyin the 1-6 Range, by Racial/Ethnic Group 



Racial/Ethnic 
t Croup 



yiW/OCR 1973 
Directory • 
Universe 
Projection* 



Principai 
Sample of 
4,750 
Schools 

■4 fi 



First-Year 
Sample'of 
243 
Schools 



Second-Year 
Sample of 

95 
Schools 



American Indian 
Asian Ampncan 
'Black 

Spanisn Heritage 
White 



OS2 
0 52 

15 22 
5 41 

'8 » 



' 091 

16 17 
5 "5 
~b 22 



0 93 

1 01 

n 55 

5 "4 
"8 ~~ 



0 77 
06> 

18.44 
5 63 

74 55 



*Thi> ximple >ysterr>dt^ai!y extruded 8 3 percent <^t the publ»t s.huol students those who are absooated vv 
small sc h<x)l ^stncts 



ith * 



percentage Percentages of other racial/ethnic groups are fairly close, being less than 2 
percentage points away ?rom the projections of the Principal Sample. The overrepresen- 
tation^of minority* students is clearly the result of the selection criteria of poverty and 
low achievement. 

Concentration of Lov*-Achte\mg Students. The percentages of schools in three catego- 
ries of percentages of students reading one grade or more below grade level were projected 
* for the three samples, as reported in Table 15-8. The second-year sarhple projections over- 
estimate the percentages of schools with larger percentages of low-achieving students. This 
* finding is an indirect result of the secondary sample-selection criterion of low achievement 

Three-Year Stability Projections were made to three categories of school stability, de- 
fined by the percentage of students v^ho remain in the school for three or more years, dis- 
counting matriculants and graduate*. From Table 15-9, it can be seen that the second-year 
sample ov^timates the percentage of the more stable (less mobility) schools by almost 
5 percent. 



Table 15-8 - 

Projected Percentages of U.S. Public Schools With GradesJn the 
1 1-6 Range, by Concentration of Lov^Achieving Students 





Percentages of Students 
Reading Below 
Grade level 


Principal »> 
Sanjple of 
4,750 ' 
^ Schools 


c 

First-Year 
Sample of 
243 
Schools 


Second -Year 
Sample of 
95 

' * Schools 




O-20 


50.8 h 


408 


- 33.9 




rr '21-50 


J 38.0 h 


48.1 


■ 50.3 




51-100 


11 2 


11 1 


i.15.8 



Table 15-9 



* Projected Percentages "of U.S. Public Schools With Grades in the 
1-6 Raifge, by Three-Year School Stability 



Three- Year Stability 
(Non-Mobility) of 
Student Body 


Principal * 
Sample of 
4,750 
Schools 


' First-Year 
Sample of ' 

243 
* Schools 


Second- Year 
Sample of 

95 
Schools 


Low (0-60% stay) 


17 42 


13 39 


_~ 13.79^^ * 


Medium .{6 1-80% slay) 


23.41 


28 72 


22.06 


*High (61-100% stay) 


59.17 


97 90 

— * 


64.15 



'2s:j 
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Grade-Span Type of School. Although schools embrace almost every conceivable com- 
bination of grades, four logical categories were defined and the percentages of schools in 
^ each category were projected from the three samples. The res.ults in Table 15-10 indicate 
that the second-year sample overestimates the percentage of full six-grade schools, a direct 
result of that exact preference in the selection of the second-year sample. 

CE funding Four ndnwndependent categories of schools were created according to 
whether or not they received certain kinds of CE funds. The comparison of percentages of 
schools in each category, presented in Table 15-11, indicates that the sefcond-year sample 
overestimates the percentage of Title I participating, schools by about 6 percent. This dif- 
ference is probably not as large as it appears, due to the different years in which the data 
were collected and the recent trend for more schools to participate in specially funded 
compensatory programs. ' < 



Table 15-10 

Projected Percentages of U.S. Public Schools With Grades in the 
V» 1-6 Range, by Grade-Span Type 



o 

Type of School 
by trade Span 

m 


Principal 
Sample of 
4,750 
Schools ' 


First-Year 
Sample of 

243 
Schools * 


Second-Year 
Sample of 

95 « 
, Schools 


Complete elementary 
feeder elementary 
Primary 

Intermediate/middle 


67.13 
12 38 . 
f 10 10 
10.39 


* 

75 05 

6 72 

9 24 / ' 
8.99 ' 


79.93 
6 56 

10.47 
2.58 












Table 15-11 






Projected* Percentages of U.S. Public Schools With Grades in 
1-6 Range, by Cotfipensatory-Education Fijnding Category 


the 


Category by 
Compensatory-Education 
Funding Source 


Principal 
Sample of 
4,750 
Schools 


First-Year 
. Sample of 
243 .\ 
Schools 


— \ 

Second-Year 
Sample of 

95 
Schools 


Title 1 funded 
Other federal funds 
State funded 
District funded , 


67 45 
28 33 
4) 82 
15 52 


73 76 
29 73 
44 70 
20 38, 


73,29 
27 64 
44 29 ' 
19.70 






V 


«. 
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CONCLUSIONS 

Based* on the comparison's* of nine characteristics, the second-year sample provides no in- m 
formation th$t would lead us to expect serious or meaningful distortions in the comparative 

' analyses. Furthermore, it is capable of providing, data' on which national prdjections could 
be based so that the projections rarely have errors above 5 percent (the ntfn-randomness of* 
the' selection would, however, prevent our calculating unbiased standard errors). In moat 
cases, the' errors or projections are in the 1 and 2 percent range. It is concluded, therefore, 
that the second year ?>ample can well be considered a representative sample and that study 

• "findings will not be significantly biased by its nature. 




■ 



CHAPTER 16. ATTRITION OF STUDENTS FROM THE 
SUSTAINING EFFECTS STUDY 



"V . . Henry J. Zagorski # 

i Lawrence A. Jordan^ x " - 

JEdward J. Colon . * , ' - * 

;„ Over a full calendar year, a stable core of 75 percent of the students in ttie Sus- 
. taining Effects Study (SES} remained enrolled in their schools, while the remaining 
25 percent moved into or outX)fthe schools. In order to determine if the attrition* 
rate migh£ be expected to bias the findings from the study,, the rates were ana. 

. lyzed for important subgroups in the study. The stable students were found to ' 
have higher achieyement scores than the atttitors, so^pure longitudinal samples 

; with not be the same as a single-shot cross-sectional sample of the nation. The ' 
meanings of relational analyses of the study will probably not be affected by this, 
but attrition rates, considered in terms vf, additional student characteristics, are 

( expected to have complex and recondite effects on comparisons amongMudents 
grouped by their participation in CE programs. . A 

Attrition is practically inevitable in a longiTirdin^lfrvesfTg^tion of the scope of SE5. It islnv 
portant to determine the extent to which attrition is related to student characteristics and 
analysis groups and whether attrition is likely to-confound the analyses, arid' interpretations 
of treatment 'effects of compensatory education. Campbell and Stanley (1966) indicate .that 
attrition (experimental mortality) should be - recognized as an important threat to study inter- * 
prejability (internal validity). They state that "mortality, lost cases, «and cases on which only", 
panjial'data are available, are troublesome to handle and are commonly swept under^he 
rug. ' This is especially tr'ue when such cases offer alternative explanations of observed 
eduCationaMreatrnent gains. Anderson {1973) criticizes current longitudinal educational 
evaluation studies for seldom providing data on the nrOmber and -type of dropouts occurr- .* 
mg, and argues that attrition represents a major "S^ce of potential error in conclusions * 
about the effects of educational treatments. } 

This study of attrition was conducted to assist readers in evaluating results and conclusions- 
*f the SES. If the students analyzed in ttie longitudinal investigation differ substantially from 
students in tfie total sample, both the internal validity of the' study and the geheralizabUity 
Of the conclusions may be qAn to question. The pflmafry objective of the investigation 
reported, here was to examine hhw students In the longitudinal pQrtion of the sample differ^ 
from students who either lefto/for to a fulT year's participation or entered the study late. 
The differences were also examined for subgroups defined by geographicS^gion, district 
size, race/ethnicity, receipt df'free'or redjjced-price meals, basic-skills achievement, and 
selection for compensatory education. * • 

Jhe'data foMhe analysis came from seve,n major instrument ;files, t providing a merged data 
file consisting of all students having data from the Comprehensive Tests of Basic Skills 
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(CTBS), Student' Background Checklist (SBC), and Compensatory Education Roster (CER), . 
from the Fall of l976toTfie Winter of 1977. During this period, tfie CTBS had been adminis- 
tered three times and the* SBC and CER were each administered twic£. From the merged 
— students were selected who were present for at least one complete CTBS administra- 
tion, were in grades 1 through 5 during theTlYst year in the 95 schools constituting the Year 
2 Representative Sample, and were not involuntary transfers between these schools and 
non-study # schools (due to school grade-structuring within some districts)/ The total' sample 
consisted of 35,808 students. The students wqre classified into six^ttrition groups described 
jr> Table 16-1, depending on the pattern of CTBS data they provided. The groups are coded 
• in the left column of Table 16-1 with 'P' for 'Present' and 'A'for 'Absent' for each test ad- 
ministration in sequence. " # „ 

t • * • 

TABULATIONS OF ACHIEVEMENT v ' 

" \ 

The PPP, PPA, PAA attrition groups in Table 16-1 have scores on the reading and math 
achievement te.sts for the Fall, 1 9 # 76 CTBS-admmistration, while the PPP, APP, and AAP attri- 
tion groups have^scores for the Fall, 1977 administration. The sixth group shown in Table 
16-1 (APA) does not have data for either of these administrations, so it was omitted from the 
■ Analysis of achievement scores. A senes^of three-group comparisons was made^or each test 
occasion, basic ski IL and cohort, using one-way*ANOVA. Preliminary analysis on a 5 per- 
cent subsample shqyved that CTBS percentile scores, probit transformations of the percen- 
tiles, and vertical-scale scores a*ll yielded essentially the same/pattern of results. Therefore, 
the analyses presented h^re are in the form of the more easily interpreted percentile scores. 
Each ANOVA was supplemented by a Duncan Multiple Range Test to examine thfe signifi- 
cance of the three pair-wise group differences (1 vs. 2, 1 vs. 3, ^nd 2w§>3). Results of the 
analysis ere reported in Table 16-2. There are. 20 ANOVAs represent! in the table (2 test 
occasions x 2 basic skills x 5' cohorts). All 20 overall F ratios are significant at the .0001 
t level. Pair-wise group differences that are not significant at th& .01 level by the Duncan Test 
are marked by connected 'bullets' (e.g., ir> cohort 4-5 for the first test administration, the 
reading and math pair-wi^e comparisons for the PPP and PPA groups are not significant). 

An examination of the paired comparisons shows that students remaining in the study gen- 
erally have higher achievement levels thap the other students across subject areas, cohorts, 
and administrations. On the average, the PPP group exceeds the PPA group by 4.1 percen- 
tile pcrf^ts (this difference *£-A4 o % where o - 28.8, as based on the average within-group 
variance), the APP group b^5L3 percentile points (.18 o ) A the AAP group by 7.5 percentile 
points (.26 a-), and the PAA^oup by 9.8 percentile points ,(.34 o ). Using Cohen's (1969)* 
criterion for evaluating the size of mean differences, these obtained difference^would be < 
consrder^d in the neighborhood of 'small' effects (.20 o ), and nof large enough for 
'medium' effects (.50^L Another commonly used criterion, although intended to be ap- 
plied to program eff^^ is that differences of ,33 o can be considered ^'educationally 
significant' (Horst, Talknadge, and Wood, 1975) and, by this criterion, the P.PP-PAA mean 
difference can be considered an educationally significant one. If treatment effects of about 
the same size are competing with these attrition effects, of course, the attrition effects may^ 
be very important as confounding factors for the analyses. 
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Attrition Groups Used for Analysis 



Croup lumbers of Percentages ■ _ . 

Code Students of Students Desfnphon 

PPP a 21,392 59.7 * Present for each of the three test administrations. 

^ A small group o( students that was^absent from* 

the second administration but present for both 
^ . the first and thjrd is also in this group, because 

their absence 1 cannot be considered attrition from* 
, the study. 

» * i 

PPA 4,150 116 ■ Absent for the third test administration only. This 

group is likely composed of students who trans- 
' * ferred ouf'between school years. However, some 

students who were still enrolled, but were absent 
.for the third^test administration in spite of make- ' 
i up testing, are in this group. 

PAA 3,115 8.7 Present for the firsTWst administration but absent \ 

front the two subsequent ones. This group con- 
• ' * • 1 si|iS;jpW ma rily of students who t/ansferred out * 

*^ dSrinjfVhe school year. However, it also includes 

* students who were absent during the spring te$ts ; 
then transferred to anbther school between 
* * school years. 

«*~ 

APP 1,707 4.8 Absent for the first test administatioh but present 

for the following two. These could be students -\ 
v * who were absent during tjje first administration 

* in spite^ of make-up testing or students who 
transferred in during the regular sctiool year. 

AAP - 4,238 1 1.8 ' ; Absent from both the first and second test ati- v 

, ministration, but present for the third. These are 
probably students who transferred inv between 

* school years. * 

- * ' ' ' •'. / - 

APA 1,206 3.4 Present for only the second test administration^ * 



.^fcdui 



Consists primarily of students who transferred 
during the regular.schoot^ye'ar, then transferred 

t again, between school years? May also in- 
clude new or old students with one or more 
absences in -spite of make-upMestjng. 



TOTALS • 35,808 ^ MOO.Of 



Er|c • f .. >' 2S8' 



i 



r Jn^ A 1 ?K .*u ' ' PAA f °, UP iS ^ nificant| y ^ than the J?PA group for all five 
cohorts Although both groups include some students who did not transfer but were merely 
absent from the test administration, the PAA-PPA comparison strongly suggests that stu- 
dents whjsyransfer out in the middle of a school year have Ibwef achievement levels than" 
those who transfer out between school years.-la the right half of Table 16-2, the comparison 
of the two groups absent for the first and second test administrations indicates few consis- 
tent differences between students who transferred in .during the school year (APP) and 
those who tans erred I in between school years (AAP). Inspection of the connected -bullets" 
in Table T6-2 also indicates that group differences are similar for reading and math scores 
and-are similar for all cohorts. • 

^The calculations performed for Table 16-2 were repeated by proportionally weighting the 
students ..n each grade according to the SES sampling cell in which they reside The 
resulting weighted means are.presented in Table 16-3. The ANOVA F-rat,bs and Duncan 



Table 16-2 



Unweighted Comparisons of Reading and Math Percentile 
Means for Attrition Croups, by Cohort 



Grade ' ^ Fa "' 1976 Di »»a . Fa ||, 1977 Da ta 

Cohort 



} * Cod£ N Readi "S Math CrouP , N ^ ^ 



* 



. 1 PPP 4,385 48 I 48 6 

'■-2 " PPtf 1,066 '44.0 ,45 3 

PAA 876 . 17 9 40.3 



PPP ' 4,380 T7*4 48.8 

2 ' 3 ' PPA ' 97< ?' '^5 % Arr ^ 46 4 45 6J 

PAA 702 '38 1 38V AAP 991 41 6 41 oV 



3-4 



PPP 4,506' * 46*6 i~ 47 4 



. _ — - - -r/ f rrr <*,5Vt> 51 1 t 50 7 f 

PPA _ 924 404 ■ 43 3 • APP 358^ 47 0 1 t 47 2 * i 

PAA . 615 , 35 9, '38 4 AAP. 808 45 6 1 .44.8 1 

PPP 4 J 38* 47 2 t * 47 9 t 

4 " 5 PPA 706 . 4 5 8 1 46 4* ,rr - Mi 440.. 43 6f 

PAA 534 37 2 '\ 40 7* . AAP 789 44.8-1 44*1 

?PP 3,483 49 6 50 3 

5,6 ' - PPA 475 441 44 »• ' ^rr- , lM 43 7 f 45 3 f 

PAA 388 57 9- .37 9 AAP " *535 45.3 1 « h 1 



' PPP 


4,885 


52 2. 


53.2 


APP 


474 


48*0 V 


' 48.4 


AAP 


1 ,115 


44,1 * 


43.2 


4 ppp 


4,m 


- 51 8 


51.8 


.„ APP 


* 352 


46 4 


45 6 


AAP 


991 


41 6 


41.9. 


PPP 


4,506 


51 1 t 


50.7 


APP 


358^ 


47 0 A 


t 4*7 2 


AAP 


808 


45 6 


1 - 4 i l 


PPP* 


4, 138 


/so 1 


48 5 


APP • 


303 


44 0*| 


43 6 


. AAP 


789 


44.8^ 


44 8 


• PPP 


3,483 


52!9 


53 Q 


APP * % 


> .220 


43.7 t 


45 3 


MR 


535 


45.3 1 


45 6 



N °' e ^ n , n ,m' ed bu "? u d V e non • SI 8 nl,IC < ln, d.fte,ences or the means lumbers of stgdents are ' 

ooseTot rflp SS ^ ad h miniS, ^ ,IOns onl V 'he PPP group Other groups and totals, o. course" are com" - 
posea of aitterent lumbers at cases . « • 

i - , 
' • • • . '« J> J 
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Tests are not applicable here since/ in effect, fhe table reflects 'population estimates' and 
the mean square error calculations ate not meaningful. Strictly.speaking, the use of the total 
grade sample weights to compute mean estimates for subpopulatiorrs within grade is not 
legitimate. Ho,wever, the procedure was not employed with the aim of making grade sub- 
population estimates.. Rather, it was simply adopted as a device to examine how the 
weights and the percentile scores wpuld interact to modify ^he observed unweighted dif- 
ferences between the attrition groups. 

Comparison of the weighted group means in Table 16-3 with the unweighted ones in Table 
16-2 across all test ^occasions, subjects, and cohorts shows that the PPP group increased by 
an average of 2.9. percentile points, the PPA group by 1.2 percentile points, and the PAA 
group by 2.3 percentile points, while the APP and AAP groups decreased slightly. Thus, 
there is a slight tendency for the overall mean differences between the PPP group and the 
other groups to increase when vyeighting is applied. 



Tabfe 16-3 



Weighted Comparisons of Reading and Math 
Percentile Means for Attrition Groups, by Cohort 



Grade 
* Cohort 


• 




Fall, 1976 Data 




> 


Fall, 1977 Data 




Group 
Code 


Reading 


Math 


Group 
t Code 


Reading 


Math 






PPP^ 


50 7 


51 3 . 


• ' PPP 


55 3 


55.0 


1-2 v 




PPA 


44 6 


47 1 


APP 


47 7 


48 4 






PAA 


"39 9 


41 8 


AAP 


45 5 


45.7 






PPP 


49.6 


52 2 


PPP 


, 55 7 


* 56.6 


2-3 




PPA 


44 2 


'47 4 


' APP 


' 47 5 


. 48 2 






PAA" 


37 1 , 


,39.2 


AAP 


43 1 


44.2 






PPP 


49 6 


50.2 


PPP 


54.3 


54 3 


3-4 




PPA 


43.0 


45 7 


APP 


45.6 


47 4 






PAA 


38.7 


40.3 


AAP 


43.4 


41 9 


ft - 




PPP 


50 4 


50.4 


PPP 


V 53 4 


51.8 


4-5 




PPA 


46 1 


47 2 


APP 


45.3 


43.9 






PAA 


40 2 


43 4 , 


AAP 


43.4 


43.4 




i 


PPR 


51 4 


52 9 


PPP 


54 4 v 


55 8 


5-6 ■ 


PPA^ 


V 42 2 


45.0 


APP 


43.9 


37 1 






PAA 




41 9 


AAP 


43 8 


44.6 



i 



TABULATIONS OF STUDENT BACKGROUND AND OTHER CHARACTERISTICS 

'The six attrition groups shown in Table 16-1 were combined into three attrition categories, 
as follows: / ' 

Attrition Category Attrition Groflps 

Departed - PPA, PAA, ^PA 

Entered - 9 APP, AAP 

Retained - PPP 

The 35,808 students (all cohorts combined) werejeross-tabulated using the attrition category 
as one dimension, and region, district size, basic-skills achievement level, race/ethnicity, 
receipt of -free or reduced-price meals, and reading and math CE participation, in turn, as 
the second dimension. The results appear in Table 16-4. 

There were small amounts of missing data' for some of the characteristics reported in Table 
16-4, due tQ incomplete responses. The cases could be assigned to attrition groups based 
. on the CTBS, but they were missing informatioti on the SBC and CER. Accordingly, for each 
missing SBC and CER characteristic^ Jthe cells withirTeach column^were .adjusted by 
distributing the cases wkh missing data in the same proportions as the cases that had'data 
available. Row percentages were then computed Using the adjusted cell frequencies. The 
chi-squjre values computed for both the adjusted and* unadjusted tables were all significant 
at the .0001 level. All the tabulations involving reading and math CE participation were fur- 
ther adjusted .to counteract the effects of school reportrng errors on the CER for CE students 
yvho left the study during the school year. These errors (am^iJities in how school cckjf-^ 
dinatbrs categprizedltudents who had already departed)' resulted in numerous CE students 
being reported as non-CE students, which artificially inflated the retention rates for the pro- 
gram categories and, deflated them for the non-program categories. (The adjustment con- 
sisted of converting the CE status data for all students who left the study during the school 
year (PAA group) into missing value indicators, and then prorating the tabulations in the 
same manner a^that described above for^students having no'£EfJ data.) 

The retention rates in Table 164 are combined and summarized in Table 16-5, which shows 
the percentages of students in the 'Retained' category within several classifications of stu- 
^dent characteristics, ranked in order from highest to lowest. 4 The percentages in Tabje 16-5 
,are not independent of one another, because students irl one set of characteristic* are also 
included in all other sets. However, the general trend of the findings is still clearly visible. 
Students in small districts have the highest retention rates (71.2 percent), while students 
residing in the Pacific region have the lowest retention rates (50.1 percent). As'expected, 
, students in large districts, students with low achievement, recipients of free on reduced- 
price meals, and minority students all tend. to have low retention rates. OtherCE partici- 
pants in Title I schools also show a relatively low retention rate. Students in the Midwest, 
Southeast, and Northeast, regular achievers, and non-recipients of free or reduced-price 
meals show relatively high retention/ rates. < 

J ' V 




* Table 1&-4 



Percentages of Students in the Representative Sample by-Attrition Category and 
by Region, District Size, Basic-Skills Achievement Level, Race/Ethnicity, 
Receipt of Free or Reduced-Price Meals, and Reading and Math CE Status 



p 




Attrition Category 




Total 








Count 






Entered 


RptAinprl 

rxviati ivu 


T 

» Region 










Northeast 


21 5 


14 6 


63.9 


1 1,207 


Southeast \ ' 


21 7 


13 4 


649 


4,255 


. Midwest * * 


22 1 


y no 


649 


' 3,480 


Central 


21 9. 


20,2 


57 9 


9,818 


* P"aoU£i— — — *""" 


31 4 


1ft 


1 


7 fldft 


District Size 










Small . " * 


1 16 1 


12 7 


71 2 


9,078 


Medium 


25.0 


17 5 


"57 5 


9,983 


Large 


27 0/ 


18 1 


54 9 


16 747 


••■^vBasic-Skills Achievement Level 




* 






Low (at or below 35th percentile) 


^ 28 1 


' 17.9 


54,0 


'14,560; 


Regular (above 35th percentile) 


20 6 


15.7 


63,7 


?1 ?4ft 


Race/Ethnicity * 




• 






Black 


25 6 


16 9 


57,5 


7,867 


Hispanic ? * 


28 7 


18 1 


53.2 


3,861 


Other 


' 22.2 


16,3 


61 5 


<t>4,080 


Free or" Reduced-Price Meals (poverty) 


** 






« 


Receive , 


273— 


17.7 


54.4 


15,320, , 


Oo not receive * * 

% 


20 5 


15.8 


63.7* 


20,488 


Reading CE Status; Students Receive ' 




t 






Title 1 plus other CE . * 


29 4 


8 1 


62.5 


706 


Title 1 only ^ 


, 23.9 


17.0 


59.1 


5J^5 • 


Other CE in Title 1 schools 


-•31 5 


13.6 ~ 


54.9 


2,334 


Other CE in other-CE schools 


28.0 


16.1 - 


55.9 


1,348 


No CE in Title 1 schools 
No CE in otherCE schools 


.20.3 


19.0 


60.7 


16,528 


26*0 


' 1 2.2 


61 8 


^ 7,048 , * 


, No CE in non-CE schools 


27 3 


' 17.4 


% 55.3 


2,649 * 


^ Math CE 'Status; Students Receive ■ . y 










-Title 1 plus other Of 


27 1 


14.6 


58.3 


4 362 


•Title 1 only 


25.0 


17.1 


57.9 


•** 3,369 


'Other CE in Title 1 schools 


32 2 


13.7 


54.1 


1,753 


Other CE in other-CE schools f 


25,5 


18.4 


56.1 


* 885 % ^ 


No CE in Title 1 schools V 


- 20,9 


/18.4 


60.7 


19,280 > 


r No CE iQ other-CE schools / 


• 26,5 


12.1 


61.4. [ 


7,510 


No CE in noh-CE schools 


27 3 


17,4 


' 55.3 ' 


2,649 


Tofal Count 


8,471 


5,945 . 


21,932 


> . 

35,808 . 


s # 
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LOG-LINEAR ANALYSES OF THE. JOINT EFFECTS OF ACHIEVEMENT, POVERTY, 
ETHNICITY, URBANISM, AND REGION ON RETENTION RATES / 

9 

The MULTIQUAL program (Bock and .Yates, 1973) was used to model retention rates as a 
function of the classifying, variables, in order to assess the joinLor combined effects of the 
variables. As shown in Table 16-5, students'in small districts and students in the Midwest, 
Southeast and Northeast regionsaend to have high retention rates. It. is of interest to know 
whether Urbanism and Region are independently related to retention or whether these par- 
ticular regions have higher retention rates because they also tend to have smaller school* 
districts. Similarly, Ta1>le 16-5 shows' that Hispanic students and students in the Pacific 
region tend to have low retention rates: If ethnicity and region are independently related to 







Table 16-5 . . . 

r 




J 




RankOrdering of Classifications of Students, by 
the Percentages of Students Retained 






Rank 


Student Classifications 

i. 


Penaentages 
Retained 


1 

' IN 


Students m small districts 
Students in the Midwest 


71.2 , ' 
64.9 




3 V 

4 * 


Students in the Southeast t 

Students in the Northeast ^ f 


64 9 

■ • 63 9 


• 


5 

6 " > 


Students not receiving tree or reduced-price meals 
j Students achieving at regular levels 


63.7 




7 
8 


Students not selected for CE services in other-CE schools, 
White and non-black or non-Hispanic students 


61.6 
61 5 




11 

t ' 

13 


Students selected for Title f and other-CE services 
Students not selected tor CE services in Title I schools _ 

# 

Srfjdents receiving £E services from Title I only w 
Students in rhe Central US 


* 64 1 
60 j 7 

^58 6 
57 9 




13 
*14 


Jiuucnib in rntfUiurTj-sizeo OiStriCtS 

Black students 


57.5 
57.5 




15 < 
r 16 


Students selected tor CE services itf other-CE" schools 
Studen|s in schools with no, CE 


56*0. 
55 3 




17 * A 

18 


, Students ih large district? 

Students selected tor other-CE services In Title 1 schools 


54.9 
54.6 




/. 19 

s» 


Students receiving free or reduced-price meals 
Studenfs achieving at lew levels 


54 4 
54.0 


r 


21 
22 


Hispanic students 4 

Students inthe Pacitic region or the U S 


53.2 
50.1 


c 
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retention, then we would predict that Hispanic students in the Pacific region would have 
particularly low retention rates. On the other hand, since almost exactly half of the Hispanic 
students are from the Pacific region (49.9 percent), we could find by an analysis of joint 
effecls that the Pacific region has low retention 'rates because of a disproportionate number 
of Hispanic students or, if region is the primary factor, (hat Hispanic students have low 
retentionirates because they happen to come disproportionately from the Pacific region. 
When we look at ethmicity and region together, as shown below, we find that none of these 
"results is obtain*?^. In fact, Hispanic students in the Pacific region tend to have natheV 
average retention rates, and we must loo^elsewhere for a basis of the low retention rates 
among Hispanic and Pacific region students. Thus, it is importanUo assess the joint Effects 
of variables, since an examination of variables one at a time can be misleading. 

Our. approach to assess the joint effect* of the variables is by means of an tt-way table 
of frequencies for, students retained and nof retained, by N-1 classifying variables. Re- 
tention rate is then considered the 'response/ factor' or dependent variable in a log-linear 
model and the /V-1 other variables are considered the 'subject .factors' or indepen- 
dent variables. > 

, • '> - 

This approach does have some disadvantages. It is not practical to model all. of the classi- 
fying variables simultaneously, because there. are limits to the size of problems handled by 
the MULTIQUAL program, and because celfsi.zes become quite smalLand the cell retertfion. 
rates become correspondingly unstable wijh too many classifying variables. Accordingly, 
we attempt^ to examine a number of cross-classifications' having three or four classifying 
variables For a first, analysis of jo.int effects, we Cammed five tables having the form: VARI- 
ABLE x COHORT x CE-CROUP x RETENTION, substituting tfce variables ACHIEVEMENT 
LOW INCOME, ETHNJCITY, URANISM and REGION, in turn, for the VARIABLE term in 
*the abotfe model. ; , 

Throughout the remainder of thi^section, we will capitalize the names of qualitative van- 
a6les,'which will have categories defined *s follows: . - , y 

ACHIEVEMENT Low— Scorecf at\*3rd Percentile or below on Total CTBS at the' first 
recorded administration. /?egu/a^-Otherwise. - s *. 

LOW INCOME: ) Poor-Recipient of free or reduced-price meals. Non-Poor-Not a 
^ . recipient. /Vf,ss/n£~SBC (basis for thrs classification)-not available. 



ETHNICITY: Bfack, Hispanic, or White/O'ther— as showrron S#C. Missing— SBC 
not available. • * • v .J" 



4 



URBAN KM* Small Mediunf, or Large district size-a surrogate availabje from sam- 
} plingVrarpe. x \ 

REC)Or)l- ' Northeast (Standard Federal Regions I, It' III), Southeast (IV), Midwest 

. 0 ' (V), Central (VI, VII, VIII), or Pacific (IX, X)~available from sampling' 
frame. 
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CE-CROUP: Title I, Other CE, or Non-Cf-Participants in Title I, Other-CE, or Non- 
CE programs, respecth/ely. (This variable was defined only in terms of. 
Reading CE, since ft wa$ felt that Math CE would yield essentially iden- 
tical results.) 

„ RETENTION: Yes- Student has valid CTBS data for fall test administrations (PPP group 
in Table 1). No-Otherwise. 

Thefariablesand their categories should be mostly^self-explanatory. Note that the students 
with missing data for LOW INCOME are also 'missing data for ETHNICITY, since -these 
variables were obtained from the -same instrument. The REGION variable was* collapsed 
from ten categories to five categories, with the aim of obtaining fewer regions having more 
equal population sizes; ten categories were felt to be too many to handle analytically using 
MULTIQUAL The ETHNICITY and CE-CROUP categories have been similarly collapsed. 
Finally, the two RETENTION categories contrast the PPP STUDENTS (retained) with all 
others (not retained). The log-linear model analyzes retention rates 'as a function of the 
classifying variables or, more precisely, they jnodel Idgits having a one-to-one relation- 
ship with the Retention tates (V 2 tn [Retained/Not Retained)) as a function of the classi- 
fying variables. , , 

ACHIEVEMENT, COHORT, AND CE-CROUP - ' . 

* 

The ACHIEVEMENT^ COHORT* CE-(?ROUP'x RETENTION analysis is typical, and will be 
discussed in detail in order to illustrate the method. 

TaW 16-6 summarizes the log : linear 'analysis of these variables. In the Janguage of -the 

* MULTIQUAL prpgram, ACHIEVEMENT, COHORT and CfeGROUP were considered as 
'subject factors,' and RETENTION as the 'response factor.' All 35,808 students were 

• classified into one of the cells of the resulting table, and for this table we^obtained an 
overall X^(30) - 604.66. (p < .001), indicating wide variations in the retention" rates for 
the cells. (If all 30 c&lls-two levels of ACHIEVEMENT, by five leveliof <?OHORT, by three 
levels of CE-CROUP-had identical retention rates, this X 2 would have been zero. The, 
high X 2 v^lue indicates a poor fit of the model requiring all cells to have identts&l rates,)- 

As shown in the top panel of Table 16-6, all but 8.9 percent, of the variation irretention 
rates is accounted for by the ttree main effects, and all but 0.9 percent by the main effects, 
, and the three, two-way interactions. Thus, we can predict retention rates quite well as a 
function of the main effects, but further precision is provided by including^one or* more of 
the two-way interactions. The three-way interaction is not needed, since X 2 (8) » 5.69, 
^which is not significant, after the two-way interactions are entered. 

Whenwe attempt to isolate the main effects and interactions that are most strongly related 
to retention rates, we have a minor problem, which is that the results depend to some ex- 
tent on the order in which the classifying variables are entered. As in a stepwise multiple 
regression analysis, thfe classifying variables tend to account for more of the variation in 
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. Table 16-6 



Log-LineaF Analysis of Retention as a Function of 
ACHIEVEMENT, COHORT, and CE-GROUP 



- 

Effect ^ 


X* 


df 


Probability 


Percent a bp 
of Yota! X 2 


• 
» 


{ 2 

Total a 








Constant • 


604.66 


• 29 ' 


< 001 


100.0 


All Mam ftif>rt* 


O "7Q 




" < .001 


8.9 


All 9-\A/3U 1 ntor3rhr»r\r I 


5.69 * 


8 


682 


0.9 


Full Model ' <>' ' * 

T 


0.00 


o 






* 1 


Parlntmn nf 








ACHIEVEMENT 


314.47-334.J8 


1 


001 


52 0-55.3 


COHORT 


189 36-202 5Q' 


4 


.001 


3J 3-33.5 


CE-GROUP ' 


22 96- 42.35 


2 


' 001 


3.8- 7.0 


ACHIEVEMENT x COHORT 


0-86- 3.69 


4 


.93O-.450 


0 1-06 


. ACHIEVEMENT x CE-GROUP # 


29.55- 30 90 


I 


001 


*9- 5.1 


COHORT x&GROUP * - 


13.75- 17.45 


8 


.089-.026 


2.3- 2.9 ' ' 


ACHIEVEMENT x COHORT x CE-GROUP 
' ~: 1 ■ — 


5 69 


8 




0.9 



rates when they are entered into the model at an early step than, when they are entered at a 
; late step. In the lower pane*! of Table 16-6, ranges of observed X 2 values are reported for all 
'possible orders of main effects and then controlling for main effects, for all possible orders 
'of two-way interactions. These ranges are usually narrow, and presumably bracket the 
'true' X 1 values. As main effects, ACHIEVEMENT, COHORT and CE-GROUP all account 
for significant variation in retention rates, with ACHIEVEMENT and COHORT together ac- 
counting for over 80 percent of the variation, and CE-GROUP accounting for~a .?huch 
smaller percentage. There is also a significant ACHIEVEMENT x CE-GROUP interaction, 
and borderlihe^COHORT x CE-GROUP interaction. The ACHIEVEMENT x COHORT Inter- 
action is the smallest effect, and is clearly not significant. 

Table 16-7 displays {he marginal tables of retention rates for all significant or near-significant 
terms in the model'. ACHIEVEMENT accounts tot the largest differences in retention rates y 
with a 10 percent difference between low and regular achievers. C6KORT is also an'im- 
portant differentiator of retention rates, which increased a linear manger from 56 percent 
fopC6hort 1-2 to 66 percent for Cohort 5-fr. Thus, low achievers and students in the lower 
cohorts tend to have lower retentioirrate9 than regular achievers and students in the higher 
cohorts. The CE-GROUP effect is smaller than the other two main effects, and shows that 
theother-CE group has l9wer retention than the Qthertwo groups, and that Title 1 group has 
slightly lower retention (2 percent low,erph*n the non-CE group. 



Table 16-7 



t Marginal Values of Retention Rates for Significant Effects in the 
ACHIEVEMENT x COHORT xCE-GROUP x RETENTION Analysis of Table 16-6 



ACHIEVEMENT 



Low 



54 



Regular 



64 



1-2 



2-3 



3-4 



4.5 



5-6 



COHORT 


56 


57 


.60 


.62 


* .66 






TiHe 1 


Other . 
CE 


Non-CE, ' 


. CE-GROUP 




59' 


.55 


'.61 








Title 1 


Other 
CE 


Non-CE 




ACHIEVEMENT 


Low 


5T 


54 


52 




x CE-GROUP 


Regular 


' 64 


• \ ' 


64 


























(Difference) 


(07) 


( 02) 


( 12) 




* 




Title 1 


■ Other* . 
CE 


. Non-CE 






1-2 


54 


52 


.57 




COHORT x 
CE-GRObP 


2- 3 - 

3- 4 

4- 5 


54 
62 
63 


€ 52 
55 
61 


.58 
61 
.62 






5*6 


64 


6* 


.67 

















The result for the main effects must b'e .qualified, however, by the results for the ACHIEVE- 
MENT x CE-GROUP interaction. ACHIEVEMENT is associated with only a^2 percent dif- 
ference in retention for the other-CE students, bgt a 7 pe'rcent difference for Title I students, 
and a 12 percerft/Merence for rton-CE students. This interaction has important implica- 
tions for an analysis of achievement gains in the longitudinal study. Iq a cemparison of Title 
I and non-CE students, for example, we will find that the Title I group has a higher retention 
ratejor low achievers that the non-CE gfaup. This differential retention rate can be con- 
sidered to be. a benefit of Title I programs, because they are apparently more successful in 
keeping low achieved in school, but it may also have the inadvertent effect of making Title I 
seem academically harmful since there are more low achievers available for retestmg in the 
Title I group This would be true, of course, if the opposite effects of regression did not 
operate and if the comparison group truly represented its national population (but see the 
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article on the reduced sample in this volume that indicates that our comparison groups are. 
slightly more biased towards low achievers^han the Title I group). Thus, if Title I and non- 
CE prograrqs had the same effect on achievement, the non-CE group would tend to show 
higher achievenrfent gains, solely as a function of differential retention rates and lower gams 
a function of regression to the mean, but would not be much'mfluenced by the sampling. 
We will return to this point again in the conclusion. * . 

The COHORT x CE-GROUP effect irt Table 16-7 can be described as follows: within each of 
the CE groups, the retention rates increase for the higher cohorts, with 'jumps' or larger 
shifts in the size of retention rates at the points indicated by horizontal lines; and the CO- 
HORT x^CE-GROUP interaction reflects the fact that these 'jumps' take place at different 
points for different CE groups. This interaction .should probably not be taken seriously, 
however, since the associated «j s only marginally^significant, and does*not reach sig- 
nificance at 'the .05 level when the othe* two-way jnteractiop terms are entered firs^J^his 
means that the effect would probably not be obtained, or would not be obtained in the 
same form, Qn replication. , * 

» 

ACHIEVEMENT, ETHNICITY, LOW INCOME, CE-CROUP, AND RETENTION 

When we repeated the analysis of the last lectin, with LOW INCOME replacing ACHIEVE- 
MENT, the resulting tables were very similar, apart from the' influence of the Missing-Data , 
category for LOW INCOME. Students missing SBC data (about 5 perce'nt of the total) were 
overwhelmingly from the PAA group, ^o thpe 'students were present for the fill CT6S ad- 
ministration and then left before other kinds of data were collated. We have no w^y of 
assigning valid LOW INCOME or ETHNICITY categories to these students, but can assdme 
that they have relatively high proportions of free or reduced-pnce lunch recipients and 
minority students. We do have information about the relationship between ACHIEVEMENT 
and LOW INCOME, as shown in Table 16-8.~ 

Thus, more low .achievers than regular achievers are missing information on the LOW IN- 
COME variable, even though only 41 .percent of students are classified as low achievers. 
There is also a fairly strong association between .LOW INCOME and ACHIEVEMENT (with 
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Cross-Tabulations of Students by ACHIEVEMENT and LOW INCOME Categories 







V 


LOW INCOME 




r~* 




V 


Poor 


Non-Poor 


Missing 


Total 


ACHIEVEMENT 


Low 
Regular 


8791 
5,628 . 


4,770 
> '14,775 


1 ( 008 
836 


14,569 
21,239 . 




Percent Low 61% 
* 


' * 24% 


55% 


41% 
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the Miising column omitted, * - .349*Tbr the remaining 2x2 table), so it#not surprising 
that,the two variables have similar relationships with RETENTION. 

• $ 

As shown m Table 16-9, popr students had lower retention rates (58 percent) than non-poor 
(67 percent), overall, but this result must be qualified by the results from the LOWJNCOME 
x CE-CROUP (able, which indicate a 3 percent di'fference between poor and non-poorjn 
the other-CE group, a "6 percent difference in the Title L group and a 10 percent difference in 
1 thq.non-CE group This is almost identical to the pattern obtained for the ACHIEVEMENT x 
CE-CROUP interaction in Table 16-7. * ' * 



r Table 16-9 

* 

Selected Marginal Tables of Retention Rates From a Log-Linear 
Analysis of LOW INCOME, COHORT, CE-CROUP and RETENTION 



LOW INCOME 

LOW INCOME 
x CE-CROUP 



Title I 
Other CE 
Non-CE 



Poor K 


Non-Poor 


^ Missing 




.67 


.01 




60 


66 


01 


57 


60 


02 


57 


.67 


.01 * 
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These results for LOW INCOME and similar results for the ETHNICITY-tfariable led to an 
attempt at predicting 'retention rates from ACHIEVEMENT, ETHNICITY, LOW INCOME and 
CE-GROUP simultaneous^. For this .analysis, stu'dents in the Missing category for LOW 
INCOME and ETHNICITY were deleted, and the data were collapsed across COHORT. The 
statistical summary fof this analysis is given in Table 16-10. 

The results*frorr? this analysis are not simple to interpret. Table 16-lOindicateS that a sfgnifi- 
cant variation in rates -is still present after all of the three-way interactions have been 
entered. Over 96 percent of the vajiation in rates" has been accounted for at that point, and 
further fitting would probably not yiefd interpretable results. In the loWer panel of Table 
1&10, we can identify effects worthy of further examination, by selecting the effects that are 
significant at the .'05 level regardless of order of entry. By this criterion, we select all four 
main effects, three of th£ two-way intejactions, and two of the three-way .interactions. The 
model containing only these effects yields X 2 (15) - 32.83, p - .005, and may be regarded 
as fitting reasonably well. The marginal valdes oP retention rates for these. effects are given 
in Table 16-11. 

In Table 16-11, wecan make certain generalization^bout retention rates from the mam- 
effect marginals: 

• Regular achievers have higher rate*' thanMow achievers. 

i , — 
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Table ,16-10 



Log-Linear Analyskof RETENTION as Function 
of ACHIEVEMENT, ETHNICITY, LOW INCOME and CE-C.ROUP 



Effect 




df 


Probability 


of Total X 2 


4 


iotai \ 




5 1 




Constant 


585 8iv 


35 


. /< 001 


100 0 


All Mam Effects 


159 62 


2? 


<-001 


27 1 


A ") . WJ .1 \ i \r\iar \f~\\r\t\c 

r\\\ ^-vvdy iiiltridvUOns 


J J. o7 


16 


^ < 001 


i 

90 




k n /o 






3 3 




- Partition of X 








ACHIEVEMENT 


106 72-241 57 - 


1 


<001 


18 2-41 2 


ETHNICITY ~ % 


13 50-104 89 


2 


'002- < 001 


2 3-179 




125 54-28i 67 


1 


<001 


21 4-48 3 




1 Q ru- rr n ■ 

lo Uo- 55 72 


* 2 


< 001 

4 


3 1- 9 S 


ACHIEVEMENT x ETHNICITY 


1 35- 4 39 • 


' 2 


: 512- 111 


0 2-08 


ACHIEVEMENT * IOW INK~OmF 


U UJ- ( £7 


i 

i 


862- 256 


0 0-02 


. ACHIEVEMENT x CE -GROUP 


9 38-' 14 72 


. 2 


009- < 001 


; 16-25 


ETHNICITY x LOW INCOME 


1 67- 6 47 


2 


434- 039 


0 3-11 


* ETHNICITY x CE-GRCXJP 


20 64U 71 28 


4 


• < 001 


3 5-12 2 


LOW INCOME x CE -CROUP ' • 


16 45-^70 70 


2 


<001 


2 8-12 1 " 


ACHIEVEMENT x ETHNICITY ,x LOW INCOME 


1266- 15 67 


2 ' 


•oo2-<oor^-' 


2 2-27 


ACHIEVEMENT x ETHNICITY x CE-CROUP 


v . T095- 12 33 


4 


027- 015 


1 9- 2.1* 


ACHIEVEMEN J x LOW INCOME x CE-CROUP ■ 


t 62- 3 13 


2 


.445-' 209 


0 3- 0.5 


ETHNICITY x LOW INCOME x CE-CROUP 


4*72- 9 67 


4 


317- 046 


0.8- 1.1 • 



* • White/other stadents have higher rates than black students, -and^l^ck students have 
higher rates than Hispanic students. ' 

• Non-poor students have higher rates than poor ones. ' 

• Non-CE students have sfightly higher rates than Title I students, and Title I $ udents 
have higherjates than other-CE students^ *) 

The ACHIEVEMENT and LOW INCOME main effects are the strongest,, and toge'therHfcey 
.account for over half of the variation in rates. 

As the interactions show, however, almost^all of the above generalizations fail to hotel for" 
one or more subgroups. 

Most of the interactions in Table 16-1 \ involve the CE-GROUP variable, and especially the 
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Table 16-11 

.u Mar 8 inal Values of Retention Rates for Significant Effects in 

the ACHIEVEMENT x ETHNICITY x LOW INCOME x CE-GROllK RETENTION 

Analysis of Table 16-10 



ACHIEVEMENT 



ETHNICITY 



LOW INCOME 



CEGROU3 

ACHIEVEMENT x CE-GROUP 



ETHNICITY x'CE-CROUP 



LOW INCOME x CE CROLP 



ACHIEVEMENT x ETHNICITY 
x LOW INCOME 

Poor 

Non-Poor 
(Poor Non-Poon 

ACHIEVEMENT x ETHNICITY 
x CE-CROUP 



Title I 
Other CE 
Non-CF 
(Title l-Non-CE) 
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1 ' 

Low 
Regular 



Black 

Hispanic 

White/Other 



Poor 1 
Non-Poor ' 



Black 



Hispanic 



White/Other 



Low v Reg^ (0\ti) Low" Reg 




Black 



Hispanic 



White/Other 



v Low 


Reg 


(Dirt) 


Low 


Reg 


(Diffr 


Low 


Reg 


(Dirt) 


63 
47 
58 
( 05) 


61 " 
46 
68 
(- 07) 


(- 02) 
C r 01) 
( 10) 
f- 12) 


56 
.61 
51 . 

t(U) 


4 66 
62 
58 
( 08) 


( 11) 
- (01) 
(07) 
(04) 


60 
58 
*58 
(•02) 


68 
62 
67 
-(01) 


1 (.08) 
(04) 

(- 01)' 




* 




i 













301 
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4 


t 


Low 


Regular 






58 


66 




Black 


Hispanic 


White/Othe 


r 


1 61 


- 57 


65 






Poor 


Non-Poor 


c 




58 


67 * 




Tifle 1 Other CE 


Non-CE 






58 t 


64 


1 




60 
66 


57 
59 


57 
67 






62 

57 / 
63 


47 
61 
60 


62 
54 

.65 






60 

65 o 


57 
59 


* 57 
67 







Other-CE subgroup. As we have seen, the other-CE students tend -to hivetaw retention 
rates, and do not show the usukl difference in rate between low and regular achievers, or 
between Door and non-poor students. When We examine the ETHNICITY x CE-GROUP 
interaction, however, we discover that the low rate for other-CE students is primarily due to 
the very low rate for the black subgroup (47 percent), and that the Hispanic, subgroup actu- 
ally has it) highest retention rate for other-CE students (61 percent. This probably indicates 
* that other-CE programs serving black stucfents are different from the other-CE programs 
serving Hispanic students. As show n in Table 16^12, it is also the case that white/other and 
black students are more hkely to teceive Title I funds, while the reverse is true for the His- 
panic group (but ^ greater proportion of Hispanic; students participate in %tle I than erther 
of the other groups). 

Table 16-12 



Cell Frequencies for ETHNICITY x CE-GROUP Cross-Classification 





Title 1 


Other CE 


Non-CE # 


Black 


1,963 


708 


. 4.777' 




Hispanic 


884 


978 


1,775 




White Other 


2,851 


1.844 


18*21.1 


















* 





These results suggest the need for caution in contrasting students reviving other-CE funds 
with other students, since the other-CE subgroup'is somewhat heterogeneous. We cannot 
say what role bilingual programs (technically not usually defined as 'compensatory/ but 
frequently considered as such by school personnel) play in thesesfrndmgs. ^ 

Three-way interactions are always difficult to characterize, and Table 16-1 1 includes two of 
them,. The ACHIEVEMENT x ETHNICITY x CE-CROUP subtable at thebottom of Table 16-1 1 
is particularly interesting. The main effect for CE-CROUP indicates that the retention rate 
for non-CE students (64 percent) is slightly higher than the rate for Title I students (62 per- 
cent). Examination of the ACHIEVEMENT x ETHNICITY x CE-CROUP marginals will show, 
however, that non-CE students have lower retention rates for five of six subgroups, and only 
black-regular achieving students show a higher retention rate for non-CE (68 percent) than 
for Title I'students (61 percent).* How ca'n that be? The apparent paradox is resolved if the 
retention rates and cell frequencies are examined simultaneously, as in Table 16-13. 

* 

Thus, the overall, rate for non-CE students is putled up by the large group of white/other ' 
regular achievers, even though non-CE students have tov^er retention rates in five of six 
columns of Table 16-13. <* 4 

We noted earlier that Title I programs retain rfiore low achievers than non-CE programs, 
resulting in a bia§ toward makitfg Title I look bad. Further complications ensue if \£/e use the 
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y Table 16-13 



Retention Rates and Cell Frequencies for 
ACHIEVEMENT xtTHNICITY x CE-GROUPSubtable 



CE-CROUP 

Title \ 

Other CE 

Non-CE 

Totals 



ETHNICITY 

M ' 1 



Black 



Hispanic 



White/Other 



Totafc 



Low 


Reg. 


Low 
i 


Reg. 


Low 


Reg. 




63 


.61 


, 55 


w .66' 


.60 , 


68. 


.68 


1,641 ~ 


325 , 


721 


164 


1,860 


990 


5,698 




1 .46 ' 


.61 


.62 


58 


,62 


.58 


N 469 


239 


686 

i 


294 


951 


892 


* 3,531 


.58 


68 


.51 


58 


.58 


" ,67 


.64 


. 2,626 


2,151 


1.004 


772 


3,614 


14,598 * 


fc 24,765 


.58 


65 


, 55 


.60 


.59 


.67 ' 




4,736 s 


2712 


2,411 


1,230 


6,425 


f6,480 





data^of Table 16-13 to compare Title I and non-CE programs within ethnic subgroups. There, 
are three different resulfs: > 

* • White/Other 'Students. Title I retains, slightly .more low achievers than non-CE (2 
percent difference), *amd alsa slightly more'segular achievers (1 percent difference)/ 
resulting in a slight but probably trfyial bias toward making Title I look bad. 

• Hibanic Students.* Title I retains more low achievers than non-CE (4 percent dif- 
0 fe* nee'),, but retains even more regular achievers (8 percent difference), resulting in a 
net bias toward making Title I look good. • * 

' * Black Students. Title I retains noticeably more low achievers than .non-CE (5 per- 
cent), and retains many fewer. regularjichievers (7 percent difference in the other 
direction), resulting in a net bias towa remaking Title I look bad. 

\ - - . , 

Thus, an apalysis of achievement gains by ethnic subgroups could have quite mischievous 
effects, showing no difference of gains fpr Title 1, or losses for Title I, solely as a function of 
differential attrition," and depending on the subgroup analyzed. 

REGION, URBANI5M, CE-GROUP, AND RETENTION 

REGION and URBAN ISM (i.e., size of school district) are both variables from the sampling 
frame far the Sustaining Effects Study. We attemped to determine their joint effects, along 
with CE-C$OJP, on RETENtlON.^The statistical summary is given in Table 16-14, and the 
relevant nlr^|ginaPvalues are given in Table 16-15. 
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Table 16-14 \ 

Log-Linear Analysis of RETENTION as a Function 
of REGION, URBANISM, and CE-CROUP 



. Effect 


X 2 


df * 


Probability 


Percentage 
of Total X 2 




v Total X 2 








Constant ^ 




A A 

44 


<.001 


100.0 


All Mam Effects 




Jo 


<,001 


23.6 


All 2-VV£y Interactions 


81.26 


16 


<.001 


* J.O 




Parhtinn nf 

rdilll \KJ\ I ui \ 






/ 


REGION % ^ 


408.26-444.86 


4 


<.ooi. ' 


28 3-30.8 


URBANISM flW* 


653.72-685.25 , 


2 


<,001 


45.3-47 5 ' 


CECROUP 


0.19- 34.88 


2 


909-<J.001 - 


00.0-2.4 


REGION x URBANISM 


BO 75-136.92 


8* 


<.ooi . 


9.1- 9.5 


REGION xCE-GROL^P 


80.26- 98.11 


8 


<.001 


5.6- 6.8 


URBANISM x CE-GlfoUP 


24 85- 41.61 


4 


<.001 


1.7- 2.9 


REGION x URBANISM x CE-GROUP 
♦ 


8K26 


-16 


<.001 


5.6* 



\ 



The largest effect is for URBANISM, which accounts for^ nearly half of the variation in rates. 
Small districts have retention rates that are roughly 15 percent* higher than medium and 
large districts, and a pure longitudinal sample will have a disproportionate number of stu- 
dents from small districts. REGION accounts for more than #guarter of the variation in rates, 
and reveals ap east/west contrast, with .relatively high rates (64-65 percent) in the Northeast, 
Southeast and Midwest regions, and lower r^tes (58 percent and 50 percent) in the Central- 
and Pacific regions. As shown by the REGION x URBANISM marginals in Table 16-15, stu- 
dents from small districts in the East tend to have the highest rates, apd students from large 
districts in ti^ Central and Pacific regions tend to have the lowest rates, which is'what we 
would e\pe«;if REGION and URBANTSM were independently related to retention. There is 
a substantial REGION x URBANISM interaction, however, which accounts for about 9 per- 
cent of the total- variation, and which appears to be attributable to atypical values for the 
Midwest-Medium cell. By taking simple deviations from the marginal rates, we obtain the 
following table of REGION x URBANISM' deviations: 





Small 


Medium 


Large 


Northeast . 


.03 


-.01 


-.01 


Southeast 


-.01 


.00 


-c.04 


Midwest 


-.04 


.24 


i 


Central 


-.04 


-.01 




Pacific 


.01 


-.01 


- .02 



/ ■ 
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• \ . Table 16-15 

' t $ Marginal Values ol Retention Rates for Significant Effects in the 

: REGION x URBANISM x CE-GROUP x RETENTION Analysis of Table 16-14 









" Small^ 


Medium 


Large 




URBANISM 






.71 


.57 


,55 . 




c(-GROUP 




» 


y Title 1 


Other CE 


Non-CE 








.60 


.55 


.60 








Northeast 


Southeast 


Midwest 


Central 


P^ficN, 


RECION 




64 


65 ' • 


.65 


58 


.5& 






A f 










a 

f 


REGION 

x Q-GROUP 


Title I 
Other CE 
Non-CE 


61 . 
.59 
•65 


.61 
' 66 


71 

< 57 
.65 


.50 
* .58 


.51 
.53 
.48 








* 












REGION 

x URBANISM 

•> 


Small 
1 Medium 
Large 


78 
61 
58 


75 
.62 
56 


.72 
.76 
.58 


.65 
54 
.55 


.63 
' 47 
.47 




¥ 






Small » 


Medium 


Larg£ 




qrbanism 

w CE-GROUP 


Title I 
Other CE 
Non-CE 




70. 
.64 
.72 


.59 
.51 - 
58 


*53 

.55 
.55 



















Thus, the rate deviation for the Midwest-Medium cell stands out since "it is 24 points higher 
than expected. In Table 16-15, the rate for the Midwest-Medium cell more closely resem- 
bles the rate for the Midwest-Small cell, while the rate for medium districts in Qther regions 
resembles the corresponding rate for large districts. In the Midwest region, the schools in 

- the medium subgrbup are from larger towns in predominantly rural areas, and have a more 
rural character than their district sizes would imply. Students. <and their families may be 
more mobile in areas served by medium and large school districts, partly^ explaining their 
lower retention rates. The consequences of parental mobility for attrition are also greater in 
suburban and urban areas, since a move of only a few miles Within a metropolitan area is 

-likely to mean a transfer to another school. , 4 

<* ' • ' -v 

ERIC . / 305 282 ' 



All of the two-way interactions in Table 16-14 are significant, but the CE-GROUP main effect 
is reduced to almost zero when REGION and URBANISM are entered first. This means that 
the CE groups do not have different retention rates overall after controlling for REGION 
and URBANISM^ There are REGION x'CE-GROUP and URBANISM x CE-GROUP interac- 
tions, however and both tables are marked primarily by unexpectedly \high rates for one of 
the subgroups of the other-CE group-for the Pacific region. in one case, and for large dis- 
tricts in the second. • \ 

The mam conclusion to be drawn from this analysis is that REGION and URBANISM effects 
are considerably larger than the CE-GROUP effects. Because REGION and URBANISM 
are variables from the sampling frame, we would expect students within categories of 
these classifying variables to be relatively homogeneous on many variables, and they do 
seem to be more homogeneous with respect to retention rates. However, there are also 
indications that students in the three CE-GROUP categories are unevenly distributed over 
the REGION and URBANISM categories are unevenly distributed over the REGION and 
URBANISM categories. , " ' 

\ fairly substantial three-way interaction remains after fitting all the two-way interaction 
terms, with x 2 (16) = 81 26, p < .001. Table 16-16 presents the three-way table of 

Table 16-16 

Retention Rates and Cell Frequencies for 
a REGION x URBANISM x CE-GROUP Subtable* * 



URBANISM 



Small 



' Medium 



Large 



NE 



SE 



J? 

3 



MW 



Total 



TitleT 


Other CE 


Non-CE 


Title 1 


Other CE 


Non-CE 


Title 1 


Other CE * Non-CE 


Total 


74 


( 74) 


80 


63 


( 53) 


61 


48 


58 


60 


64 


596 


120 


2.102 


507 


232 


1.818 


819 


844 


4,139 


11.197 


f 7?f 


( 64) 


77 


J 59 




64 


56 




56 


65 


244 


159 # 


1.113 


261 


11 


1,081 


445 


9 


940 


4.262 


1 77) 




< 71) 






76 




_ % 


59 


.65 


188 


3 


. 362 


41 


54 


80O 


63 


85 


1.894 


3.469 


65 




' 66 


57 


( 53) 


54 


63 


1 49f 


54 


58 


436 


80 


2,407 


556 


159 


2.15J, 


519 


282 


3,222 


9.811 


f 58) 


( 63) 


64 


< 46) 


( 42) 


48 


51 


54* 


37 


50 


156 


no 


1,009 


194 


235 


1.868 


933 


1/338 j 


1.206 ' 


7.046 


* 70— 64 


72 


58 


51 


55 


53 


, 54 


— ; J 
55 




1,620 


472 


6,994 


1,559 


690 


7.739 


2,777 


2.557 


1>.400 





'Rates omitted tor cells with \ <J00 and enclosed in parentheses fQr cells with 100<N<400 ' 
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retention rates and cell frequencies. This table has. a feature not encountered in earlier 
tables, namely, that some of the cell frequencies are too small for reliable estimate* of the 
retention rates. We have omitted rates for cells having frequencies of 100 or below, and 
.rates for^ cells having frequencies in the 100-400 range are enclosed in parentheses, as a 
. reminder that these rates have relatively large standard errors (from .025 to about .05). 
Judging from the cell N$, other-CE programs affect very few students outside of large 
districts in the Northeast and Pacific regions. Almost 60 percent of other-CE Mudents^re in 
these two cells. Title I students are much more widely distributed. The three-way interac- 
tion for predicting rates may owe more to sampling fluctuations than do any substantive in- 
fluence. Both the highest and lowest rates are for Non-CE students, with the highest rate in 
the Northeast-small-noh-CE cell (80 percent), and the lowest rate in the Pacific-large-non- 
CE cell (37 percent).^ % . % 

To help clarify the REGION x URBANISM x CE-GROUP results, analyses were undertaken 
in which the variables ACHIEVEMENT and ETHNICITY were adde^J to these three variables. 
These further analyses provided more detail, but scant clanficatroh. Adding more variables 
seems to make it more difficult rather than less difficuJt to achieve acceptable fits to the 
data. Small cell sizes, together with sampling variability and the unreliability of the fclassify- 
" ing variables, may be adding noise faster than information. - r 

Table 16^17 



Retention Rates <tnd Cell Frequencies for 
a REGION x URBAN ISM x ETHNICITY Subtable* 















4 • 

URBANISM 
















Small 






Medium 






Large 










Black 


Hispanic 


White/ 
Other 


C 

■ Black 


..Hispanic 


White/ 
Other 


Black 


Hispanic 
J. ^ 


White/ 
Other 


Total 




NE 


r 

75 


^28 


80 
2,660 


73* 

Am 


54 
745 


64 
1,260 


58 
2,518 


. 45 
*47T 


69 
2,450 


67 
10,694. 


z 




(82) 




V 7 


,67 




68 


68 




55 


68 . 


ECIO 


'232' 


9 


U29 


■ 498 


9 


744 


425 


18 


881 


4,045 




MW 


19 ' 


^74 
3 


521 


6 


10 


79 . 
848 


54 

739 


16 - 


* 67, 
1,159 


68 

3.320 




C 


( 76) 




67 


( 58) 


(62) ; 


57 


6< 


( 48) 


56 


61 - 


V 






39 


2.580 


259 f 


382 


2,040^ 


' 1.111 


109 


2,645 


9.361 




P" 




(67) - 


• 69 




(53) 


50 


k 50 


59 


43 


54 




23 


394 


758 


26 


240 


1,894 


8,444 


1,184 


1,204 


6,567 














> 














Total 


79 


67 


74 


67 o 




61 


57 


✓ 55 


59 






545. 


- 472 


7,747 


'1,276 


1,386 


6,786 


5,637 


1.798 


8,340 





•Rates omitted for cells <100, and enclosed in parentheses for cells with 100<N<400 
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For its intrinsic interest, a REGION i URBANISM x EHTNICITY subtable from these further 
analyses is given in Table 16-17. Again, the rates for cells with Ns smaller than 100 omitted, 
and those for cells with /Vs between 100 and 400 are enclosed in parentheses. Examination 
of the Ns' indicates that except for, Hispanic students in the Pacific region, there are 
very few minority students in small districts, with large districts having the most minority 
students, especially for blacks. Eighty-three percent of Hispanic students are in either the 
Northeast or Pacific regions, and another 15 'percent are,in the Central region, leaving a 
mere 2 percent in the Southeast and Midwest regions. As indicated earlier, nearly ha4f of 
the Hispanic students are in the Pacific region, and they have an overall retention rate of 
, .60, which is about average. Hispanic students rn the Northeast region' have ^somewhat 
lower retentioa rate (.51). Most Hispanic: ^students in the Pacific region probably have a 
Mexican or Central American background, while most Hispanic students in the Northeast 
probably have a Puerto Rican or other Caribbean background. The, differences in attrition 
rate could indicate that analyses invplving Hispanic students should distinguish between 
those in eastern and western parts of the country. Hispanic students }n. large" Northeast 
districts have one of the lowest retention rates in the table (.45). The lowest rate of all 
belongs to white/other students in large Pacific districts, however, (.43), which is surprising. 
If we think of attrition as a risk factor, wAwould expect minority students to show higher 
attrition. In three of five comparisons for wge districts, white/other students have lower 
retention than black students. 

SUMMARY AND CONCLUSIONS N < 

This study extends the description of the SE$> representative sample into the second year, 
with special emphasis on identifying the demographic factors most closely associated with 
student attrition and pn assessing the impact of such factors on longitudinal comparisons 
among educational programs. A total of 35,808 students, tested at some time during the 
first year of study, 'were classified by attendance (attrition) pattern, geographic region, ur- 
banism, race7ethnicity, receipt of free or reduced-pric£ meals, basic-skills achievement, 
cohort, and selection for compensatory education. The .results of these classifications were 
examined to assess the main and joint effects on retention and then to interpret such effects 
in terms of the objectives of the longitudinal study. 

Attrition is widespread in the nation's schools, which should come as no surprise. In indi- 
vidual schools, pupil retention rates ranged from a low of 28 percent to a high of 87 percent. 
Overall, only about 60 percent of the students could be tested in the fall of two successive 
years. About 20 percent of the students tested at Timfe I departed before the close of the 
study, and another 20 percent entered the study after Time 1. This,way of stating retention 
rates, as a percentage of the number oftstudents passing through the study, teryls to exag- 
gerate attrition to some extent. With these figures, one could rephrase the findings so .that 
there is a relatively stable core consisting or 75 percent of the students, while 25 percent of 
the students move into and out of the studyjn any given year. However, changing fhe base 
for the rate would not affect conclusions aboubdifferential attrition for various subgroups. 

We assume that most of y the observed attrition occurs when students and their families 
move into or away from the study schools. We can distinguish tl?ree cases: (1) students 



move because of a permanent change of residence, (2) students are temporarily lost to the' 
study,* because of illneSs, staying with a relative, and so on, (3) students are in the schools, 
bc/t their dataware lost to the study through clerical error, unscorable tests, and other kinds 
of data collection problems. Case*(3) is the only source of attrition which is under our con- 
trol. Students with a PAP (Present-Absent-Present) pattern are interesting, because while we 
cannot distinguish between the three cases in the data, case (1) is an unlikely reason ^for a 
PAP daja pattern-^that is, we don't expect marty instances where students will make a per- 
manent change of residence twice in a year, moving'away from and then back to the same 
school district. The number of PAP students is small, however, and represents less than 1 
percent s of the students having Achievement test data at any one time. This implies that data * 
collection problems are not a major cause of the observed attrition for the SES. i 

For purposes of analysis, the sipjelgstmodels for longitudinal data will be those that use 
-'pure' longitudinal samples,y^si5ting*^ with complete data on all variables. 

This study shows that such /sample will not belfi^aQie as any cross-sectional one. There 
are models for data with missing observations, (Dut'they have limifaJtions"for the present are 
models for data with missing observations, but they have limitations for the present study. 

Afifi and Elashoff (196$) and Tim^(1970) have surveyed the relevant literature. Morrison 
states the usual caveat. "Of course, in every_case it jis essential that the causes of the.missing 
data are completely independent of the nature or values of the response vanates'.' (Morn- " 
son, 1976, pg 120) But, of course, in non-experimental research tbi^ caveat vVTiJ^lmost in- 
variably be violated. When the response variate is achievement, it is easy to think of casual 
relationships with missmg data. On the one hand, movement of families can disrupt chil- 
dren's educations, and lead to lower achievement, on the other hand, low achievement 
may cause some families to transfer their children to different and perhaps better school^. 

There is ample evidence that the values for achievement scores are higfier for studegts hav- 
ing stable PPP data patterns than for students many of the attrition groups. What this shows 
is that stable' students are differeht from the students who enter or leave, <£[igd thus "do not 
represent students generally. That may pot be a serious problem, since students entering or 
leaving the sample are, by defmifion, not receiving the full benefit of their CE or other edu- 
cational programs To -evaluate a program fairly, we might in pnncjpfe want to limit the 
analyses to students receiving the program for at least a full year.tA more serious statistical 
problem is the problem of differential attrition, where different subgroups have different ' 
rates*qf attrition. The most striking instance of differential attrition was obtained for a Title 
I non-CE comparison wTthm ethnic' subgroups. Other things .being equal, and disregarding 
the opposing effects of regression^Vhite/other students had little differential attrition, His- 
panic students had differential attrition which could help Title I look good and black stu- 
dents had differential attrition which could make'Title I look bad. Becausfe a Smaller number ' 
of Hispanic students receive Title 1 funds, the.net effect of these patterns would be to bias a 
pure longitudinal sample in the direction of making Title I look less effective in tompansorts* 
wrth' non-CE programs if regression and sampling effects did not completely counteract the 
bias. Although it is anyone's guess which exerts a greater influence on the findings, it $^ms 
important' to incorporate these concerns into the analysis of longitudinal data, even it only » 



{ * ■ ^ ; ; 

by caveat. It is our general opinion that our analyses will be'influenced by several mihiscule, 
biases/ but that they work in opposite directions' so that we do not have to be concerned 
that they will gang-up and seriously distort our findings. The biases are further recluced'or* 
eliminated entirely for analyses that use more carefully selected groups than merely non-CE 
studerfts to serve as the' comparisons, t * - *■ 
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